Vera Clemens (Sep 22 2023 at 14:13): Vera Clemens (Sep 22 2023 at 14:13):Hi,
I'm working on a PR https://github.com/IQSS/dataverse/issues/9926 and wanted to add and run a test, but I'm having trouble starting my Dataverse locally... When I run mvn -Pct package docker:run, it fails with the following error:
[ERROR] Failed to execute goal io.fabric8:docker-maven-plugin:0.43.0:build (default-build) on project dataverse: Unable to build image [gdcc/configbaker:unstable] : "The command '/bin/sh -c true && apk add --no-cache ${APK_PACKAGES} && mkdir -p ${SCRIPT_DIR} ${SECRETS_DIR} ${SOLR_TEMPLATE}' returned a non-zero code: 8" -> [Help 1]
I have the current develop branch checked out.
Could someone help me troubleshoot?
Philip Durbin 🚀 (Sep 22 2023 at 14:15):Absolutely. Let me pull the latest from develop (3b7d824) and check.
Philip Durbin 🚀 (Sep 22 2023 at 14:19):@Vera Clemens actually, as a sanity check, could you please add "clean"? Like this:
mvn -Pct clean package docker:run
I worked for me without the "clean". Hmm. I wonder what's different. :thinking:
Philip Durbin 🚀 (Sep 22 2023 at 14:23):Something else to try, building just the configbaker image:
mvn -Pct docker:build -Ddocker.filter=dev_bootstrap
Are you on Mac, Linux or Windows?
Vera Clemens (Sep 22 2023 at 14:24):I tried running it with clean before, same error. And building the configbaker image results in the same error as well.
Vera Clemens (Sep 22 2023 at 14:24):I am on Linux
Vera Clemens (Sep 22 2023 at 14:25):Hmm, if it works for you, it must be some kind of issue with some cache or something on my end?
Philip Durbin 🚀 (Sep 22 2023 at 14:28):There's some docker command like system prune that blows away the cache. Not sure if it will help.
Philip Durbin 🚀 (Sep 22 2023 at 14:28):When you try to build just configbaker, is there any more output? Any clues?
Vera Clemens (Sep 22 2023 at 14:35):Just ran docker rmi $(docker images -a --filter=dangling=true -q) and docker rm $(docker ps --filter=status=exited --filter=status=created -q) to try and clear any caches (found here: https://forums.docker.com/t/how-to-delete-cache/5753/2) but no change
Vera Clemens (Sep 22 2023 at 14:36):The full output is:
~/Documents/dataverse$ mvn -Pct docker:build -Ddocker.filter=dev_bootstrap
[INFO] Scanning for projects...
[INFO]
[INFO] ----------------------< edu.harvard.iq:dataverse >----------------------
[INFO] Building dataverse 6.0
[INFO] ----------------------------[ docker-build ]----------------------------
[INFO]
[INFO] --- docker-maven-plugin:0.43.0:build (default-cli) @ dataverse ---
[INFO] Reading assembly descriptor: /home/clemens/Documents/dataverse/modules/container-configbaker/assembly.xml
[INFO] Copying files to /home/clemens/Documents/dataverse/target/docker/gdcc/configbaker/unstable/build/maven
[INFO] Building tar: /home/clemens/Documents/dataverse/target/docker/gdcc/configbaker/unstable/tmp/docker-build.tar
[INFO] DOCKER> [gdcc/configbaker:unstable] "dev_bootstrap": Created docker-build.tar in 180 milliseconds
[ERROR] DOCKER> Unable to build image [gdcc/configbaker:unstable] : "The command '/bin/sh -c true && apk add --no-cache ${APK_PACKAGES} && mkdir -p ${SCRIPT_DIR} ${SECRETS_DIR} ${SOLR_TEMPLATE}' returned a non-zero code: 8" ["The command '/bin/sh -c true && apk add --no-cache ${APK_PACKAGES} && mkdir -p ${SCRIPT_DIR} ${SECRETS_DIR} ${SOLR_TEMPLATE}' returned a non-zero code: 8" ]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 14.375 s
[INFO] Finished at: 2023-09-22T16:35:24+02:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal io.fabric8:docker-maven-plugin:0.43.0:build (default-cli) on project dataverse: Unable to build image [gdcc/configbaker:unstable] : "The command '/bin/sh -c true && apk add --no-cache ${APK_PACKAGES} && mkdir -p ${SCRIPT_DIR} ${SECRETS_DIR} ${SOLR_TEMPLATE}' returned a non-zero code: 8" -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
It's Feierabend time now, thank you for your help so far. Will keep trying on Monday :)
Philip Durbin 🚀 (Sep 22 2023 at 14:40):Ha, I've heard of Feierabend but for Monday. Maybe you can edit modules/container-configbaker/Dockerfile and try splitting up the following onto different RUN lines:
RUN true && \
# Install necessary software and tools
apk add --no-cache ${APK_PACKAGES} && \
# Make our working directories
mkdir -p ${SCRIPT_DIR} ${SECRETS_DIR} ${SOLR_TEMPLATE}
Have a good weekend!
Vera Clemens (Sep 25 2023 at 08:48):Hope you had a good weekend as well!
Vera Clemens (Sep 25 2023 at 08:48):That narrows it down to:
[ERROR] Failed to execute goal io.fabric8:docker-maven-plugin:0.43.0:build (default-cli) on project dataverse: Unable to build image [gdcc/configbaker:unstable] : "The command '/bin/sh -c apk add --no-cache ${APK_PACKAGES}' returned a non-zero code: 8" -> [Help 1]
Vera Clemens (Sep 25 2023 at 08:49):(btw: what is the purpose of the true &&?)
Vera Clemens (Sep 25 2023 at 10:11):More clues: I just ran $ docker build . --build-arg SOLR_VERSION=8.3.0 in modules/container-configbaker and this is the output:
Sending build context to Docker daemon 24.06kB
Step 1/18 : ARG SOLR_VERSION
Step 2/18 : FROM solr:${SOLR_VERSION} AS solr
---> 17dcf43eea9e
Step 3/18 : FROM alpine:3
---> c1aabb73d233
Step 4/18 : ENV SCRIPT_DIR="/scripts" SECRETS_DIR="/secrets" SOLR_TEMPLATE="/template"
---> Using cache
---> 45d9286c2a05
Step 5/18 : ENV PATH="${PATH}:${SCRIPT_DIR}" BOOTSTRAP_DIR="${SCRIPT_DIR}/bootstrap"
---> Using cache
---> 36e58a2ed9a8
Step 6/18 : ARG APK_PACKAGES="curl bind-tools netcat-openbsd jq bash dumb-init wait4x ed"
---> Using cache
---> 4a2b261b3f13
Step 7/18 : RUN true
---> Using cache
---> 625bfdb14a4d
Step 8/18 : RUN apk add --no-cache ${APK_PACKAGES}
---> Running in fd628057a6d8
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/APKINDEX.tar.gz
WARNING: fetching https://dl-cdn.alpinelinux.org/alpine/v3.18/main: temporary error (try again later)
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/APKINDEX.tar.gz
WARNING: fetching https://dl-cdn.alpinelinux.org/alpine/v3.18/community: temporary error (try again later)
ERROR: unable to select packages:
bash (no such package):
required by: world[bash]
bind-tools (no such package):
required by: world[bind-tools]
curl (no such package):
required by: world[curl]
dumb-init (no such package):
required by: world[dumb-init]
ed (no such package):
required by: world[ed]
jq (no such package):
required by: world[jq]
netcat-openbsd (no such package):
required by: world[netcat-openbsd]
wait4x (no such package):
required by: world[wait4x]
The command '/bin/sh -c apk add --no-cache ${APK_PACKAGES}' returned a non-zero code: 8
I ran the docker build command again with --network host and the installation no longer failed. So, there seems to be some network connectivity issue during the mvn docker:build...
Vera Clemens (Sep 25 2023 at 11:01):Success: $ DOCKER_HOST=unix:///var/run/docker.sock mvn -Pct clean package docker:run fixed the problem. But still the Dataverse doesn't start:
dev_bootstrap> 2023-09-25T10:27:06Z INF [HTTP] Checking the http://dataverse:8080/api/info/version ...
dev_bootstrap> 2023-09-25T10:27:06Z ERR Expectation failed error="the status code doesn't expect" actual=404 expect=200
dev_bootstrap> 2023-09-25T10:27:14Z INF [HTTP] Checking the http://dataverse:8080/api/info/version ...
dev_bootstrap> 2023-09-25T10:27:14Z ERR Expectation failed error="the status code doesn't expect" actual=404 expect=200
dev_bootstrap> 2023-09-25T10:27:22Z INF [HTTP] Checking the http://dataverse:8080/api/info/version ...
dev_bootstrap> 2023-09-25T10:27:22Z ERR Expectation failed error="the status code doesn't expect" actual=404 expect=200
dev_bootstrap> 2023-09-25T10:27:30Z INF [HTTP] Checking the http://dataverse:8080/api/info/version ...
dev_bootstrap> 2023-09-25T10:27:30Z ERR Expectation failed error="the status code doesn't expect" actual=404 expect=200
dev_bootstrap> 2023-09-25T10:27:38Z INF [HTTP] Checking the http://dataverse:8080/api/info/version ...
dev_bootstrap> 2023-09-25T10:27:38Z ERR Expectation failed error="the status code doesn't expect" actual=404 expect=200
dev_bootstrap> 2023-09-25T10:27:46Z INF [HTTP] Checking the http://dataverse:8080/api/info/version ...
dev_bootstrap> 2023-09-25T10:27:46Z ERR Expectation failed error="the status code doesn't expect" actual=404 expect=200
dev_bootstrap> 2023-09-25T10:27:54Z INF [HTTP] Checking the http://dataverse:8080/api/info/version ...
dev_bootstrap> 2023-09-25T10:27:54Z ERR Expectation failed error="the status code doesn't expect" actual=404 expect=200
dev_bootstrap> 2023-09-25T10:28:02Z INF [HTTP] Checking the http://dataverse:8080/api/info/version ...
dev_bootstrap> 2023-09-25T10:28:02Z ERR Expectation failed error="the status code doesn't expect" actual=404 expect=200
dev_bootstrap> 2023-09-25T10:28:10Z INF [HTTP] Checking the http://dataverse:8080/api/info/version ...
dev_bootstrap> 2023-09-25T10:28:10Z ERR Expectation failed error="the status code doesn't expect" actual=404 expect=200
dev_bootstrap> 2023-09-25T10:28:18Z INF [HTTP] Checking the http://dataverse:8080/api/info/version ...
dev_bootstrap> 2023-09-25T10:28:18Z ERR Expectation failed error="the status code doesn't expect" actual=404 expect=200
dev_bootstrap> 2023-09-25T10:28:26Z INF [HTTP] Checking the http://dataverse:8080/api/info/version ...
dev_bootstrap> 2023-09-25T10:28:26Z ERR Expectation failed error="the status code doesn't expect" actual=404 expect=200
dev_bootstrap> 2023-09-25T10:28:34Z INF [HTTP] Checking the http://dataverse:8080/api/info/version ...
dev_bootstrap> 2023-09-25T10:28:34Z ERR Expectation failed error="the status code doesn't expect" actual=404 expect=200
dev_bootstrap> 2023-09-25T10:28:42Z INF [HTTP] Checking the http://dataverse:8080/api/info/version ...
dev_bootstrap> 2023-09-25T10:28:42Z ERR Expectation failed error="the status code doesn't expect" actual=404 expect=200
dev_bootstrap> 2023-09-25T10:28:50Z INF [HTTP] Checking the http://dataverse:8080/api/info/version ...
dev_bootstrap> 2023-09-25T10:28:50Z ERR Expectation failed error="the status code doesn't expect" actual=404 expect=200
dev_bootstrap> 2023-09-25T10:28:58Z INF [HTTP] Checking the http://dataverse:8080/api/info/version ...
dev_bootstrap> 2023-09-25T10:28:58Z ERR Expectation failed error="the status code doesn't expect" actual=404 expect=200
dev_bootstrap> 2023-09-25T10:29:06Z INF [HTTP] Checking the http://dataverse:8080/api/info/version ...
dev_bootstrap> 2023-09-25T10:29:06Z ERR Expectation failed error="the status code doesn't expect" actual=404 expect=200
dev_bootstrap> Error: context deadline exceeded
What if you increase the timeout from 3m in modules/container-configbaker/scripts/bootstrap.sh?
Philip Durbin 🚀 (Sep 25 2023 at 12:51):Also, @Oliver Bertuch probably has some ideas. And can probably explain why he added the "true" you mentioned. :grinning: I found this: https://stackoverflow.com/questions/73301757/run-true-in-dockerfile
Vera Clemens (Sep 25 2023 at 13:08):Trying with an increased timeout right now... But after looking more closely at the log, I think this may be the problem:
dev_dataverse> [#|2023-09-25T10:27:00.705+0000|SEVERE|Payara 5.2022.4|javax.enterprise.system.core|_ThreadID=46;_ThreadName=RunLevelControllerThread-1695637590671;_TimeMillis=1695637620705;_LevelValue=1000;|
dev_dataverse> Exception while loading the app|#]
dev_dataverse>
dev_dataverse> [#|2023-09-25T10:27:00.729+0000|SEVERE|Payara 5.2022.4|javax.enterprise.system.core|_ThreadID=46;_ThreadName=RunLevelControllerThread-1695637590671;_TimeMillis=1695637620729;_LevelValue=1000;|
dev_dataverse> Exception while loading the app : Class edu.harvard.iq.dataverse.settings.spi.AliasConfigSourceProvider has unsupported major or minor version numbers, which are greater than those found in the Java Runtime Environment version 11.0.18|#]
I see, interesting :) (about the true)
Oliver Bertuch (Sep 25 2023 at 13:22):@Vera Clemens are you running on Java 17? It sure doesn't look like...
Vera Clemens (Sep 25 2023 at 13:32):@Oliver Bertuch hmm maybe not, I ran update-java-alternatives to switch in case I wasn't, now java -version prints "openjdk version "17.0.8.1" 2023-08-24", but the above error remains the same
Vera Clemens (Sep 25 2023 at 13:33):Sorry, I know very little about Java :face_in_clouds:
Vera Clemens (Sep 25 2023 at 13:34):Shouldn't the version of Java used be set by the Dockerfiles somehow? The one used inside the container at least? Not sure why the error message says something about Java version 11
Oliver Bertuch (Sep 25 2023 at 13:55):You are definitely not using an up to date base image - I see Payara 5.2022.4 in there and Java 11. So if you compile with 17, you'll be in trouble.
Oliver Bertuch (Sep 25 2023 at 13:55):Please go ahead and re-pull the base image from the hub
Vera Clemens (Sep 26 2023 at 06:23):Ahh I see, thanks!
Vera Clemens (Sep 26 2023 at 06:24):That is looking better but still not loading. Here's the error messages I see now:
SOLR:
dev_solr> 2023-09-26 06:18:56.818 ERROR (coreContainerWorkExecutor-2-thread-1) [ ] o.a.s.c.CoreContainer Error waiting for SolrCore to be loaded on startup => java.util.concurrent.ExecutionException: org.apache.solr.common.SolrException: Unable to create core [collection1]
Flyway:
dev_dataverse> [#|2023-09-26T06:19:54.148+0000|SEVERE|Payara 6.2023.8|javax.enterprise.system.core|_ThreadID=50;_ThreadName=RunLevelControllerThread-1695709140312;_TimeMillis=1695709194148;_LevelValue=1000;|
dev_dataverse> Exception while loading the app : jakarta.ejb.CreateException: Initialization failed for Singleton StartupFlywayMigrator
dev_dataverse> jakarta.ejb.CreateException: Initialization failed for Singleton StartupFlywayMigrator
...
dev_dataverse> Caused by: org.flywaydb.core.api.exception.FlywayValidateException: Validate failed: Migrations have failed validation
dev_dataverse> Detected resolved migration not applied to database: 5.13.0.1.
dev_dataverse> To ignore this migration, set -ignoreMigrationPatterns='*:ignored'. To allow executing this migration, set -outOfOrder=true.
dev_dataverse> Detected resolved migration not applied to database: 5.13.0.2.
dev_dataverse> To ignore this migration, set -ignoreMigrationPatterns='*:ignored'. To allow executing this migration, set -outOfOrder=true.
dev_dataverse> Need more flexibility with validation rules? Learn more: https://rd.gt/3AbJUZE
...
dev__dataverse> [#|2023-09-26T06:19:54.155+0000|WARNING|Payara 6.2023.8|fish.payara.boot.runtime.BootCommand|_ThreadID=50;_ThreadName=RunLevelControllerThread-1695709140312;_TimeMillis=1695709194155;_LevelValue=900;|
dev_dataverse> Boot Command deploy failed PlainTextActionReporterFAILUREDescription: deploy AdminCommandError occurred during deployment: Exception while loading the app : jakarta.ejb.CreateException: Initialization failed for Singleton StartupFlywayMigrator. Please see server.log for more details.
dev_dataverse> [name=dataverse
dev_dataverse>
dev_dataverse>
dev_dataverse> PER01003: Deployment encountered SQL Exceptions:
dev_dataverse> PER01000: Got SQLException executing statement "CREATE TABLE EXTERNALTOOLTYPE (ID SERIAL NOT NULL, TYPE VARCHAR(255) NOT NULL, EXTERNALTOOL_ID BIGINT NOT NULL, PRIMARY KEY (ID))": org.postgresql.util.PSQLException: ERROR: relation "externaltooltype" already exists
dev_dataverse> PER01000: Got SQLException executing statement "CREATE INDEX INDEX_EXTERNALTOOLTYPE_externaltool_id ON EXTERNALTOOLTYPE (externaltool_id)": org.postgresql.util.PSQLException: ERROR: relation "index_externaltooltype_externaltool_id" already exists
dev_dataverse> PER01000: Got SQLException executing statement "CREATE TABLE DATAVERSETHEME (ID SERIAL NOT NULL, BACKGROUNDCOLOR VARCHAR(255), LINKCOLOR VARCHAR(255), LINKURL VARCHAR(255), LOGO VARCHAR(255), LOGOALIGNMENT VARCHAR(255), LOGOBACKGROUNDCOLOR VARCHAR(255), LOGOFOOTER VARCHAR(255), LOGOFOOTERALIGNMENT INTEGER, LOGOFOOTERBACKGROUNDCOLOR VARCHAR(255), LOGOFORMAT VARCHAR(255), TAGLINE VARCHAR(255), TEXTCOLOR VARCHAR(255), dataverse_id BIGINT, PRIMARY KEY (ID))": org.postgresql.util.PSQLException: ERROR: relation "dataversetheme" already exists
dev_dataverse> PER01000: Got SQLException executing statement "CREATE INDEX INDEX_DATAVERSETHEME_dataverse_id ON DATAVERSETHEME (dataverse_id)": org.postgresql.util.PSQLException: ERROR: relation "index_dataversetheme_dataverse_id" already exists
...
OK, fixed those errors with sudo rm -r docker-dev-volumes, now it's running :) I ran docker volume prune before, which somehow didn't do the trick... Thanks a lot for your help!
Vera Clemens (Sep 26 2023 at 07:20):PR is posted :+1: https://github.com/IQSS/dataverse/pull/9956
Philip Durbin 🚀 (Sep 26 2023 at 11:22):Fantastic! Thanks for sticking with it!
Philip Durbin 🚀 (Sep 26 2023 at 11:22):So what were the biggest problems getting the dev environment up? Not having the required version of Java? Anything else?
Philip Durbin 🚀 (Sep 26 2023 at 11:23):You also needed this, which I don't seem to need on Mac:
DOCKER_HOST=unix:///var/run/docker.sock mvn
Yes, I think in summary the fixes to my issues were 1. setting the DOCKER_HOST (otherwise the apk add in the Dockerfile fails due to no network connectivity), 2. pulling the most current version of the base image, and 3. removing the old postgres data in docker-dev-volumes (I assume those were leftovers from a dev environment I had running month(s) ago).
Philip Durbin 🚀 (Sep 26 2023 at 11:33):Oh, interesting, did you have a base image saved from a few weeks (or months) ago? Before we released Dataverse 6.0 (which uses Payara 6 in the base image).
Vera Clemens (Sep 26 2023 at 11:34):Yes, pretty sure I did.
Philip Durbin 🚀 (Sep 26 2023 at 11:36):Ok. Any thoughts on how to document this? Maybe right after the quickstart at https://guides.dataverse.org/en/6.0/developers/dev-environment.html we could say "If you have any trouble, please see the troubleshooting section."
And "troubleshooting" would be a link, of course.
Vera Clemens (Sep 26 2023 at 11:58):Hmm, I just re-checked whether the DOCKER_HOST actually fixed the network issue and apparently not. It was something else.
I noticed because I reverted the change of splitting the RUN command into three RUNs, and the network error re-occured even with the DOCKER_HOST set.
I guess it only appeared as if setting the DOCKER_HOST fixed the issue because the real fix allowed the result of the apk add to be cached.
Here's what the real fix was: running docker build modules/container-configbaker/ --build-arg SOLR_VERSION=8.3.0 --network host, then re-running the mvn command.
Tbh this seems kind of hacky to me. @Oliver Bertuch do you have an idea how this network issue should be properly fixed?
Vera Clemens (Sep 26 2023 at 12:00):Philip Durbin said:
Ok. Any thoughts on how to document this? Maybe right after the quickstart at https://guides.dataverse.org/en/6.0/developers/dev-environment.html we could say "If you have any trouble, please see the troubleshooting section."
And "troubleshooting" would be a link, of course.
That sounds good. :+1: Once we have the above thing properly figured out I can try and draft something and submit it as a PR if you'd like
Vera Clemens (Sep 26 2023 at 12:30):Adding <network>host</network> to the dev_bootstrap image in pom.xml (see https://dmp.fabric8.io/#build-configuration) fixes the issue for me, but not sure if that breaks anything for anybody else...
Oliver Bertuch (Sep 26 2023 at 13:45):To me, all of this sounds like some local specialty in your Docker setup interfering with the build. Something like the host mode for network is really weird. Are you doing this on Win, Linux or Mac?
Philip Durbin 🚀 (Sep 26 2023 at 13:46):Linux (I already asked).
Philip Durbin 🚀 (Sep 26 2023 at 13:46):Maybe we should say, "If you have any trouble, come tell us in chat.dataverse.org!" :grinning:
Oliver Bertuch (Sep 26 2023 at 13:51):I'm on Linux as well and don't need any of these specials. Today I did have a problem with my Docker not being able to pull images when coming from Maven, but was when doing it via CLI
Oliver Bertuch (Sep 26 2023 at 13:52):Solution was to restart the Docker daemon - looks like the DNS server was cached and could not be reached any longer.
Vera Clemens (Sep 26 2023 at 14:18):Yes. Linux inside a VM though, maybe that changes things
Philip Durbin 🚀 (Oct 20 2023 at 14:50):Related: mvn -Pct clean package docker:run error #10011
Balázs Pataki (Oct 28 2023 at 12:41):I just got the same error. I tried the docker install like 4 month ago and so I had old base images. After removing and reloading the images, now it works.
I wonder if there maybe a way for the docker:run target to check whether the currently available base image is compatible with the rest of the build (Java version, Dataverse version, etc)?
Philip Durbin 🚀 (Oct 30 2023 at 13:40):Hmm, I'm reminded of this PR by @Oliver Bertuch feat(ct): add labels with service dependency versions to images #9930
Not sure if it would help or not but the app image now at least knows what versions of PostgreSQL and Solr it need.
Last updated: Nov 01 2025 at 14:11 UTC