Philip Durbin 🚀 (Nov 03 2023 at 14:34): Philip Durbin 🚀 (Nov 03 2023 at 14:34):Huh. I wonder why I'm seeing this all of the sudden:
dev_dv_initializer> chown: /dv: I/O error
Context? Reproducer? More log output?
Oliver Bertuch (Nov 03 2023 at 14:35):This rather general message can be like a myriad of things... :shrug:
Philip Durbin 🚀 (Nov 03 2023 at 14:35): Oliver Bertuch (Nov 03 2023 at 14:36):Gnrf I hate Zulip for not showing this to me right away. Have to download first. A builtin pastebin thing would be great
Philip Durbin 🚀 (Nov 03 2023 at 14:36):$ mvn -Pct docker:run
[INFO] Scanning for projects...
[INFO]
[INFO] ----------------------< edu.harvard.iq:dataverse >----------------------
[INFO] Building dataverse 6.0
[INFO] ----------------------------[ docker-build ]----------------------------
[INFO]
[INFO] --- docker-maven-plugin:0.43.4:run (default-cli) @ dataverse ---
[INFO] DOCKER> [gdcc/configbaker:unstable] "dev_bootstrap": Skipped running
[INFO] DOCKER> [gdcc/configbaker:unstable] "dev_bootstrap": Start container 80a6f0dd0a71
dev_bootstrap> Waiting for http://dataverse:8080 to become ready in max 3m.
dev_bootstrap> 2023-11-03T14:32:51Z INF [HTTP] Checking the http://dataverse:8080/api/info/version ...
dev_bootstrap> 2023-11-03T14:32:51Z ERR Expectation failed error="the status code doesn't expect" actual=404 expect=200
[INFO] DOCKER> [gdcc/configbaker:unstable] "dev_dv_initializer": Start container 7182f144683e
dev_dv_initializer> chown: /dv: I/O error
dev_dv_initializer> chown: /dv: I/O error
[INFO] DOCKER> [postgres:13] "dev_postgres": Start container 6cf276243147
[ERROR] DOCKER> Error occurred during container startup, shutting down...
[INFO] DOCKER> [gdcc/configbaker:unstable] "dev_dv_initializer": Stop and removed container 7182f144683e after 0 ms
[INFO] DOCKER> [gdcc/configbaker:unstable] "dev_bootstrap": Stop and removed container 80a6f0dd0a71 after 0 ms
[ERROR] DOCKER> (At least one exception was thrown while trying to remove custom networks.)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.909 s
[INFO] Finished at: 2023-11-03T10:32:51-04:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal io.fabric8:docker-maven-plugin:0.43.4:run (default-cli) on project dataverse: (At least one exception was thrown while trying to remove custom networks.) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
Sorry, I was going back and forth about uploading a text file or pasting. There you go ^^
Oliver Bertuch (Nov 03 2023 at 14:38):To me this looks like something is wrong on your machine
Oliver Bertuch (Nov 03 2023 at 14:38):Disk full?
Philip Durbin 🚀 (Nov 03 2023 at 14:38):So my machine crashed yesterday. Then I went home.
Oliver Bertuch (Nov 03 2023 at 14:38):Urgh. Can you please delete the volume dirs? There might be some dangling fs thing going on
Philip Durbin 🚀 (Nov 03 2023 at 14:39):disk is not full:
Screenshot-2023-11-03-at-10.38.53-AM.png
Philip Durbin 🚀 (Nov 03 2023 at 14:40):I already ran rm -rf docker-dev-volumes but I can do it again.
Philip Durbin 🚀 (Nov 03 2023 at 14:40):I also ran docker system prune and restarted Docker.
Philip Durbin 🚀 (Nov 03 2023 at 14:40):I'm running out of tricks. :sweat_smile:
Oliver Bertuch (Nov 03 2023 at 14:41):Sorry but is Docker on Mac using a VM or not?
Philip Durbin 🚀 (Nov 03 2023 at 14:41):Good question. Not sure.
Philip Durbin 🚀 (Nov 03 2023 at 14:42):I'm trying docker system prune -a
Oliver Bertuch (Nov 03 2023 at 14:44):Yes, Docker Desktop does use a Linux VM on Mac
Oliver Bertuch (Nov 03 2023 at 14:45):Maybe take a look around if nothing comes up in the management interface? https://gist.github.com/BretFisher/5e1a0c7bcca4c735e716abf62afad389
Philip Durbin 🚀 (Nov 03 2023 at 14:50):HMDC-beamish:dataverse pdurbin$ docker run -it --rm --privileged --pid=host justincormack/nsenter1
/ # df -h
Filesystem Size Used Available Use% Mounted on
tmpfs 393.4M 468.0K 392.9M 0% /run
tmpfs 393.4M 0 393.4M 0% /tmp
tmpfs 1.9G 0 1.9G 0% /var
dev 1.9G 0 1.9G 0% /dev
shm 1.9G 0 1.9G 0% /dev/shm
cgroup_root 10.0M 0 10.0M 0% /sys/fs/cgroup
tmpfs 1.9G 8.0K 1.9G 0% /containers/onboot/000-dhcpcd/tmp
tmpfs 1.9G 0 1.9G 0% /containers/onboot/001-sysfs/tmp
tmpfs 1.9G 0 1.9G 0% /containers/onboot/002-sysctl/tmp
tmpfs 1.9G 0 1.9G 0% /containers/onboot/003-format/tmp
tmpfs 1.9G 0 1.9G 0% /containers/onboot/004-extend/tmp
tmpfs 1.9G 0 1.9G 0% /containers/onboot/005-mount/tmp
/dev/vda1 58.4G 22.7G 32.7G 41% /var/lib
tmpfs 1.9G 0 1.9G 0% /containers/onboot/006-metadata/tmp
tmpfs 1.9G 0 1.9G 0% /containers/onboot/007-services0/tmp
tmpfs 1.9G 0 1.9G 0% /containers/onboot/008-services1/tmp
tmpfs 1.9G 0 1.9G 0% /containers/onboot/009-swap/tmp
tmpfs 1.9G 0 1.9G 0% /containers/onboot/010-mount-docker/tmp
/dev/vda1 58.4G 22.7G 32.7G 41% /containers/services
/dev/vda1 58.4G 22.7G 32.7G 41% /var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/docker
/dev/vda1 58.4G 22.7G 32.7G 41% /containers/services/docker
tmpfs 1.9G 0 1.9G 0% /containers/onboot/011-bridge/tmp
tmpfs 1.9G 4.0K 1.9G 0% /containers/services/acpid/tmp
tmpfs 1.9G 4.0K 1.9G 0% /var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/acpid/tmp
overlay 1.9G 4.0K 1.9G 0% /containers/services/acpid/rootfs
overlay 1.9G 4.0K 1.9G 0% /var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/acpid/rootfs
tmpfs 1.9G 0 1.9G 0% /containers/services/binfmt/tmp
tmpfs 1.9G 0 1.9G 0% /var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/binfmt/tmp
overlay 1.9G 0 1.9G 0% /containers/services/binfmt/rootfs
overlay 1.9G 0 1.9G 0% /var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/binfmt/rootfs
tmpfs 1.9G 8.0K 1.9G 0% /containers/services/dhcpcd/tmp
tmpfs 1.9G 8.0K 1.9G 0% /var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/dhcpcd/tmp
overlay 1.9G 8.0K 1.9G 0% /containers/services/dhcpcd/rootfs
overlay 1.9G 8.0K 1.9G 0% /var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/dhcpcd/rootfs
tmpfs 1.9G 0 1.9G 0% /containers/services/diagnose/tmp
tmpfs 1.9G 0 1.9G 0% /var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/diagnose/tmp
overlay 1.9G 0 1.9G 0% /containers/services/diagnose/rootfs
overlay 1.9G 0 1.9G 0% /var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/diagnose/rootfs
tmpfs 1.9G 0 1.9G 0% /containers/services/dns-forwarder/tmp
tmpfs 1.9G 0 1.9G 0% /var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/dns-forwarder/tmp
overlay 1.9G 0 1.9G 0% /containers/services/dns-forwarder/rootfs
overlay 1.9G 0 1.9G 0% /var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/dns-forwarder/rootfs
tmpfs 1.9G 628.0K 1.9G 0% /containers/services/docker/tmp
tmpfs 1.9G 628.0K 1.9G 0% /var/lib/mount-docker-cache/entries/docker.tar/70918e63c378be8683de4fc10c553798322a67941c518075202eab14ca0a8c55/containers/services/docker/tmp
tmpfs 1.9G 628.0K 1.9G 0% /var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/docker/tmp
overlay 1.9G 628.0K 1.9G 0% /containers/services/docker/rootfs
overlay 1.9G 628.0K 1.9G 0% /var/lib/mount-docker-cache/entries/docker.tar/70918e63c378be8683de4fc10c553798322a67941c518075202eab14ca0a8c55/containers/services/docker/rootfs
overlay 1.9G 628.0K 1.9G 0% /var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/docker/rootfs
tmpfs 1.9G 0 1.9G 0% /containers/services/http-proxy/tmp
tmpfs 1.9G 0 1.9G 0% /var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/http-proxy/tmp
overlay 1.9G 0 1.9G 0% /containers/services/http-proxy/rootfs
overlay 1.9G 0 1.9G 0% /var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/http-proxy/rootfs
tmpfs 1.9G 0 1.9G 0% /containers/services/kmsg/tmp
tmpfs 1.9G 0 1.9G 0% /var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/kmsg/tmp
overlay 1.9G 0 1.9G 0% /containers/services/kmsg/rootfs
overlay 1.9G 0 1.9G 0% /var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/kmsg/rootfs
tmpfs 1.9G 0 1.9G 0% /containers/services/procd/tmp
tmpfs 1.9G 0 1.9G 0% /var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/procd/tmp
overlay 1.9G 0 1.9G 0% /containers/services/procd/rootfs
overlay 1.9G 0 1.9G 0% /var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/procd/rootfs
tmpfs 1.9G 0 1.9G 0% /containers/services/sntpc/tmp
tmpfs 1.9G 0 1.9G 0% /var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/sntpc/tmp
overlay 1.9G 0 1.9G 0% /containers/services/sntpc/rootfs
overlay 1.9G 0 1.9G 0% /var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/sntpc/rootfs
tmpfs 1.9G 0 1.9G 0% /containers/services/socks/tmp
tmpfs 1.9G 0 1.9G 0% /var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/socks/tmp
overlay 1.9G 0 1.9G 0% /containers/services/socks/rootfs
overlay 1.9G 0 1.9G 0% /var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/socks/rootfs
tmpfs 1.9G 0 1.9G 0% /containers/services/trim-after-delete/tmp
tmpfs 1.9G 0 1.9G 0% /var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/trim-after-delete/tmp
overlay 1.9G 0 1.9G 0% /containers/services/trim-after-delete/rootfs
overlay 1.9G 0 1.9G 0% /var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/trim-after-delete/rootfs
[snip]
I had to cut off the last few lines. More of the same. tmpfs overlay stuff.
Oliver Bertuch (Nov 03 2023 at 14:51):That does not look like something is full...
Oliver Bertuch (Nov 03 2023 at 14:51):Can you run a filesystem check?
Oliver Bertuch (Nov 03 2023 at 14:52):Both for the VM and for you local filesystem?
Oliver Bertuch (Nov 03 2023 at 14:52):What happens if you just start a container e.g. using Alpine, bind mount something and start to chown?
Philip Durbin 🚀 (Nov 03 2023 at 14:52):Buh. Maybe. Should I get a new laptop?
Oliver Bertuch (Nov 03 2023 at 14:53):Oh the crash wasn't the software it was a FHCE? (Flying Hardware Crash Event)
Philip Durbin 🚀 (Nov 03 2023 at 14:53):Yeah, kernel panic or something.
Oliver Bertuch (Nov 03 2023 at 14:54):Yikes. Yeah this kind of thing sure can make a filesystem go awry
Philip Durbin 🚀 (Nov 03 2023 at 14:54):Can you cook me up an Alpine chown docker compose? It will take me forever. Or give me a one liner?
Oliver Bertuch (Nov 03 2023 at 14:54):One sec
Philip Durbin 🚀 (Nov 03 2023 at 14:58):/ # cat /etc/issue
Welcome to LinuxKit
## .
## ## ## ==
## ## ## ## ## ===
/"""""""""""""""""\___/ ===
{ / ===-
\______ O __/
\ \ __/
\____\_______/
my fsck fu is rusty
/ # fsck /
fsck from util-linux 2.34
fsck: error 2 (No such file or directory) while executing fsck.ext2 for /
mkdir -p foobar; docker run -it --rm -v "$(pwd)/foobar:/test" alpine:latest "chown" "1001:1001" "/test"
Philip Durbin 🚀 (Nov 03 2023 at 15:01):$ mkdir -p foobar; docker run -it --rm -v "$(pwd)/foobar:/test" alpine:latest "chown" "1001:1001" "/test"
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
96526aa774ef: Already exists
Digest: sha256:eece025e432126ce23f223450a0326fbebde39cdf496a85d8c016293fc851978
Status: Downloaded newer image for alpine:latest
Did it chown proper? (ls -n)
Philip Durbin 🚀 (Nov 03 2023 at 15:04):$ ls -nd foobar
drwxr-xr-x@ 2 503 0 64 Nov 3 11:01 foobar
Sorry, not fully understanding what's going on. I'm the slow kid.
Philip Durbin 🚀 (Nov 03 2023 at 15:08):Here, how about this? I think the chown worked:
$ mkdir -p foobar; docker run -it --rm -v "$(pwd)/foobar:/test" alpine:latest sh -c "chown 1001:1001 /test; ls -ldn /test"
drwxr-xr-x 2 1001 1001 64 Nov 3 15:01 /test
Here's a chown to a different user (nobody). Also seems to work:
$ mkdir -p foobar; docker run -it --rm -v "$(pwd)/foobar:/test" alpine:latest sh -c "chown 65534:65534 /test; ls -ldn /test"
drwxr-xr-x 2 65534 65534 64 Nov 3 15:01 /test
I clicked "uninstall" Docker Desktop and it's been trying for a long time. 20 minutes? Just a white screen:
Screenshot-2023-11-03-at-12.26.29-PM.png
Philip Durbin 🚀 (Nov 03 2023 at 16:49):This gets better and better:
Screenshot-2023-11-03-at-12.48.48-PM.png
Philip Durbin 🚀 (Nov 03 2023 at 16:50):I couldn't uninstall Docker. So I downloaded the latest version and installed it. Now I can't even configure it. :disappointed:
Something with network stuff?
Philip Durbin 🚀 (Nov 03 2023 at 16:51):"Uninstalling Docker Desktop from the command line is not available. As a workaround, you can uninstall Docker Desktop from the Dashboard." -- https://docs.docker.com/desktop/release-notes/#4250
Philip Durbin 🚀 (Nov 03 2023 at 16:52):Via https://github.com/docker/for-mac/issues/6598
Philip Durbin 🚀 (Nov 03 2023 at 16:52):I can't even get to the dashboard now. So I can't uninstall.
Philip Durbin 🚀 (Nov 03 2023 at 16:56):Error: {"message":"installing vmnetd: retrieving vmnetd version from /Library/LaunchDaemons/com.docker.vmnetd.plist: reading plist /Library/LaunchDaemons/com.docker.vmnetd.plist: open /Library/LaunchDaemons/com.docker.vmnetd.plist: no such file or directory"}
Philip Durbin 🚀 (Nov 03 2023 at 16:57): Philip Durbin 🚀 (Nov 03 2023 at 16:59): Philip Durbin 🚀 (Nov 03 2023 at 17:08):Ha ha! I was able to get to the uninstall button anyway. And this time it worked! I hope:
Screenshot-2023-11-03-at-1.00.56-PM.png
Philip Durbin 🚀 (Nov 03 2023 at 17:10):I rebooted just to be safe. Then installed Docker.
Philip Durbin 🚀 (Nov 03 2023 at 17:10):No vmnetd error when doing setup this time.
Philip Durbin 🚀 (Nov 03 2023 at 17:11):dev_bootstrap> Done, your instance has been configured for development. Have a nice day!
Philip Durbin 🚀 (Nov 03 2023 at 17:11):\o/
Philip Durbin 🚀 (Nov 03 2023 at 17:11):it works!
Philip Durbin 🚀 (Nov 03 2023 at 18:29):Woof, another kernel panic:
Philip Durbin 🚀 (Nov 09 2023 at 14:17):Thankfully, I haven't had another kernel panic. Resolving.
Notification Bot (Nov 09 2023 at 14:17):Philip Durbin has marked this topic as resolved.
Last updated: Oct 30 2025 at 05:14 UTC