Juan Pablo Tosca Villanueva (Nov 03 2023 at 13:28): Juan Pablo Tosca Villanueva (Nov 03 2023 at 13:28):I am working on submitting a PR today with the .gitattributes file to better support windows by removing the need to configure [core.autocrlf]. But after this there is a couple of steps required that are not "natural" or "normal" behavior that would be amazing if we can fix them but otherwise, we should mention the workaround on the documentation? @Philip Durbin
As I mentioned earlier on #9949 the first time that the payara container launches (On windows) it will launch without the application deployed, this will cause the configbaker to fail since the API will not be available, so we need to add probably information about this and the following instructions:
1) Restart the dataverse container (I do this from docker desktop if anyone can provide a CLI command we can include it)
2) Run the configbaker-1 container (Same as above)
these 2 steps are only required on the first launch and can be avoided if we figure out why payara is missing the application deployed at the beginning, I tested this didn't happened to me on MAC.
Juan Pablo Tosca Villanueva (Nov 03 2023 at 13:45):I am also thinking that we need to add a step to configure the host file to include the dataverse to 127.0.0.1
Philip Durbin 🚀 (Nov 03 2023 at 14:08):Interesting. At a high level I think we should start with better documentation and additional commands, if necessary.
Philip Durbin 🚀 (Nov 03 2023 at 14:08):Longer term it would be nice to figure out what the additional Payara restart is necessary, for example.
Philip Durbin 🚀 (Nov 03 2023 at 14:23):Any details on why the war file isn't deployed? Logs or whatever?
Oliver Bertuch (Nov 03 2023 at 14:30):Philip Durbin said:
Any details on why the war file isn't deployed? Logs or whatever?
Plz note it's not a WAR file but an exploded WAR. This might make some difference.
Oliver Bertuch (Nov 03 2023 at 14:31):@Juan Pablo Tosca Villanueva is this reproducible in some way?
Juan Pablo Tosca Villanueva (Nov 03 2023 at 14:42):Oliver Bertuch said:
Juan Pablo Tosca Villanueva is this reproducible in some way?
Yes, you need to checkout the project and start the containers, the very first time Payara starts without any application running, need to manually restart the container.
Juan Pablo Tosca Villanueva (Nov 03 2023 at 14:52):
Juan Pablo Tosca Villanueva (Nov 03 2023 at 14:54):We can join in a zoom call if you both want to look, it shouldn't take more than a few minutes to checkout, compile and start the project. This issue seems to be consistent and I have tested this so far on Win 10 and Win 11
Philip Durbin 🚀 (Nov 03 2023 at 14:56):I'd love a zoom call but I've having my own problems right now in #containers > dev_dv_initializer> chown: /dv: I/O error
Oliver Bertuch (Nov 03 2023 at 15:04):Juan Pablo Tosca Villanueva said:
We can join in a zoom call if you both want to look, it shouldn't take more than a few minutes to checkout, compile and start the project. This issue seems to be consistent and I have tested this so far on Win 10 and Win 11
I have a few. https://fz-juelich-de.zoom.us/j/69596667050?pwd=STd5UE1adG1oaTFsYnFMV1RRL2NzUT09
Philip Durbin 🚀 (Nov 03 2023 at 15:14):Postgres, why you so slow on Windows?!?
Screenshot-2023-11-03-at-11.13.41-AM.png
Philip Durbin 🚀 (Nov 03 2023 at 15:15):https://stackoverflow.com/questions/65285379/docker-volume-mapping-windows-incredible-slow
Philip Durbin 🚀 (Nov 03 2023 at 15:19):Let's add the fancy wait for: https://github.com/IQSS/dataverse-frontend/commit/d97f917677db3cc0a201f35af467088d2092f823
Philip Durbin 🚀 (Nov 03 2023 at 15:19): depends_on:
dev_solr_initializer:
condition: service_completed_successfully
https://github.com/compose-spec/compose-spec/blob/master/spec.md#long-syntax-1
Philip Durbin 🚀 (Nov 03 2023 at 15:23):https://github.com/peter-evans/docker-compose-healthcheck#waiting-for-postgresql-to-be-healthy
Philip Durbin 🚀 (Nov 03 2023 at 15:32):Just capturing this comment from Oliver: 'The alternative would be adding " wait4x tcp postgres:5432" to the dv_init container'
Oliver Bertuch (Nov 03 2023 at 16:04):Bummer - the Docker Maven Plugin does not properly support the long depends_on syntax (and probably also doesn't have the backing code to support the described construction). See https://github.com/fabric8io/docker-maven-plugin/issues/888
Oliver Bertuch (Nov 03 2023 at 16:04):Plan for now is to add waiting in the init script of the app container using the base image shipped "wait-for" script.
Philip Durbin 🚀 (Nov 03 2023 at 16:17):wait-for postgres:5432 -t 120 at the end of src/main/docker/scripts/init_2_configure.sh seems to work!
Juan Pablo Tosca Villanueva (Nov 03 2023 at 17:20):Ok I sent the PR
Oliver Bertuch (Nov 11 2023 at 07:47):My PR to DMP was just merged. So at least we can use the long syntax once released.
Oliver Bertuch (Nov 11 2023 at 07:48):I will have to refactor more to enable adding healthchecks though, so we are not completely there yet for the Postgres problem.
kuhlaid (Nov 29 2023 at 15:59):I'm still trying to find a containerize build setup for development Dataverse. Something I tried on Windows yesterday was to clone the Dataverse code, change to the dataverse directory in the code, and run docker run -it --name myDvContainer -v "$(pwd)":/usr/src/mymaven -w /usr/src/mymaven maven:3.8.5-openjdk-17 mvn -Ddocker.skip=true -Pct clean package. This built the dataverse-6.0.jar from the source but I had to add -Ddocker.skip=true to the Docker run command because I was getting errors with the fabric8 maven plugin. Anyway, this is not a good approach in my opinion. Dataverse probably needs something like the Docker configuration used by https://github.com/datahub-project/datahub/tree/master/docker. I would love to help in this effort, but I know very little about the Dataverse configuration so figuring out the Docker commands is beyond my expertise.
Philip Durbin 🚀 (Nov 29 2023 at 16:38):@kuhlaid sorry to hear you're having trouble. I appreciate that you're willing to help. Even just testing is extremely valuable!
Philip Durbin 🚀 (Nov 29 2023 at 16:39):Just so I'm clear, you tried mvn -Pct clean package docker:run per https://guides.dataverse.org/en/6.0/developers/dev-environment.html#quickstart FIRST? And when that didn't work you tried the docker run you posted?
Juan Pablo Tosca Villanueva (Nov 29 2023 at 18:40):Hi @kuhlaid ! I run Dataverse on Windows (10 & 11) and the command that @Philip Durbin provided should complile the project, create the images and start the containers. If you have any issues please let us know. :)
Philip Durbin 🚀 (Nov 29 2023 at 18:48):@kuhlaid we (JP and I) just read your comment at https://github.com/IQSS/dataverse/issues/10148#issuecomment-1832202521 and I think we understand what you're trying to do. You don't want to install Java or Maven locally, so you're trying to run Maven from inside the container. Hmm. :thinking:
Philip Durbin 🚀 (Nov 29 2023 at 18:51):@kuhlaid you could try this docker compose -f docker-compose-dev.yml up
Philip Durbin 🚀 (Nov 29 2023 at 18:51):That's from https://guides.dataverse.org/en/6.0/container/dev-usage.html#running
kuhlaid (Nov 29 2023 at 20:32):Thank you @Philip Durbin , but when I run the compose command I receive: Error response from daemon: error creating temporary lease: write /var/lib/desktop-containerd/daemon/io.containerd.metadata.v1.bolt/meta.db: read-only file system: unknown.
Philip Durbin 🚀 (Nov 29 2023 at 20:32):Whoa! Weird.
Philip Durbin 🚀 (Nov 29 2023 at 20:33):A suggestion to clear space from Docker and restart it: https://stackoverflow.com/a/70071216/19464
kuhlaid (Nov 29 2023 at 20:40):Yep, needed to clear my Docker space. This looks promising. Building in progress.
kuhlaid (Nov 29 2023 at 20:49):It seems like the hangup now is initdb: error: could not change permissions of directory "/var/lib/postgresql/data": Operation not permitted on the Docker compose.
kuhlaid (Nov 29 2023 at 20:51):The postgres container keeps restarting because it is unable to change permissions on the directory.
Philip Durbin 🚀 (Nov 29 2023 at 20:53):@Juan Pablo Tosca Villanueva do you think this could be related to what you said in your initial post on this topic, that on Windows you have to stop and start the Dataverse container?
Philip Durbin 🚀 (Nov 29 2023 at 20:57):@kuhlaid in between runs you might want to delete the docker-dev-volumes directory. Until we get this working. That's where the data is stored.
Juan Pablo Tosca Villanueva (Nov 29 2023 at 21:11):I don't think this is related, I have never seen this exact same error but I had issues with all the tests that require the volumes but still need to do some research on that.
Juan Pablo Tosca Villanueva (Nov 29 2023 at 23:42):I just made a test deleting al my containers and images, downloading the project from zero and not running java/maven. It worked on my machine ( :rolling_on_the_floor_laughing: ). I would probably just start from zero and deleting everything and do the following:
Juan Pablo Tosca Villanueva (Nov 29 2023 at 23:42):
Juan Pablo Tosca Villanueva (Nov 29 2023 at 23:43):I posted this GIF showing the process on my machine if it is helpful by any means.
kuhlaid (Nov 30 2023 at 01:05):Back to the docker compose -f docker-compose-dev.yml up, I changed the volume on postgres to be - ./init.sql:/docker-entrypoint-initdb.d/init.sql and created an init.sql file at the root directory (just for testing) with the contents of CREATE USER docker;
CREATE DATABASE docker;
GRANT ALL PRIVILEGES ON DATABASE docker TO docker;. After making this change I was able to compose Docker and get output from http://localhost:8080/api/info/version. :)
Juan Pablo Tosca Villanueva (Nov 30 2023 at 02:50):Sorry, I edited my steps. 4 is docker compose -f docker-compose-dev.yml up
Philip Durbin 🚀 (Nov 30 2023 at 03:27):@kuhlaid if you want to talk more tomorrow, we're having a meeting: #containers > weekly meeting
Philip Durbin 🚀 (Dec 08 2023 at 15:57):@kuhlaid I just left this comment: https://github.com/IQSS/dataverse/issues/10160#issuecomment-1847436497
Can you please take a look?
I'm trying to figure out if I should delete the draft issue we created or not. Thanks!
Oliver Bertuch (Dec 14 2023 at 07:39):I also left a comment about how Dataverse work WRT table init and why you should not let the database write into overlayfs
Philip Durbin 🚀 (May 16 2024 at 19:59):@Juan Pablo Tosca Villanueva any new info on running Dataverse in Docker on Windows?
@Victoria Lubitch just asked about it: https://groups.google.com/g/dataverse-community/c/ZDKWf2p42vs/m/iElIWg67AwAJ
Juan Pablo Tosca Villanueva (May 16 2024 at 20:24):I haven't used it in a long time but I can confirm this is in effect happening :sad:
Juan Pablo Tosca Villanueva (May 16 2024 at 20:24):
Philip Durbin 🚀 (May 16 2024 at 20:33):Ha, is a custom emoji? :crazy:
Juan Pablo Tosca Villanueva (May 16 2024 at 20:34):I don't remember lol
Juan Pablo Tosca Villanueva (May 16 2024 at 20:35):I just typed :sa and :sad-docker showed up lol
Philip Durbin 🚀 (May 16 2024 at 20:36):I feel a bit helpless when in comes to Windows since I don't have a Windows box.
Philip Durbin 🚀 (May 16 2024 at 20:36):So when you say this is happening, you mean "it seems like war file is not deploying properly"?
Juan Pablo Tosca Villanueva (May 16 2024 at 20:42):I am running a new test deleting all my old images and volumes
Juan Pablo Tosca Villanueva (May 16 2024 at 20:43):I will report soon
Juan Pablo Tosca Villanueva (May 16 2024 at 20:50): Juan Pablo Tosca Villanueva (May 16 2024 at 20:51):PER01000: Got SQLException executing statement "ALTER TABLE fileaccessrequests ADD CONSTRAINT FK_fileaccessrequests_DATAFILE_ID FOREIGN KEY (DATAFILE_ID) REFERENCES DVOBJECT (ID)": org.postgresql.util.PSQLException: ERROR: constraint "fk_fileaccessrequests_datafile_id" for relation "fileaccessrequests" already exists|#]
Philip Durbin 🚀 (May 16 2024 at 20:53):I'm confused. Your log doesn't have any SQL errors.
Philip Durbin 🚀 (May 16 2024 at 20:55):What's going on here? 2024-05-16 16:46:02 /opt/payara/scripts/entrypoint.sh: line 15: /opt/payara/config/pre-boot-commands.asadmin: No such file or directory :thinking:
Juan Pablo Tosca Villanueva (May 16 2024 at 20:55): Juan Pablo Tosca Villanueva (May 16 2024 at 20:56):Sorry, it was incomplete
Juan Pablo Tosca Villanueva (May 16 2024 at 20:57):Wasn't this what we fixed with the gitproperties? :thinking:
Juan Pablo Tosca Villanueva (May 16 2024 at 20:57):mmm I have to check a bit more what is going on lol but def is an issue now
Philip Durbin 🚀 (May 16 2024 at 20:58):And with wait-for, I thought. I'm looking again at https://github.com/IQSS/dataverse/pull/10094
Philip Durbin 🚀 (May 16 2024 at 21:01):@Juan Pablo Tosca Villanueva I guess what I'm wondering is if you and @Victoria Lubitch are having the same problem. No offense, but we should fix her problem first. :grinning:
Juan Pablo Tosca Villanueva (May 16 2024 at 21:02):Prio over Rocky?
Juan Pablo Tosca Villanueva (May 16 2024 at 21:02):
Juan Pablo Tosca Villanueva (May 16 2024 at 21:05):So
Juan Pablo Tosca Villanueva (May 16 2024 at 21:06):It is WILD
Juan Pablo Tosca Villanueva (May 16 2024 at 21:06):I started at 2024-05-16 16:46:02 and it didn't finished launching until 2024-05-16 17:05:47
Juan Pablo Tosca Villanueva (May 16 2024 at 21:08):The application is deployed and itt says running but can't reach it
Juan Pablo Tosca Villanueva (May 16 2024 at 21:08):neither the API
Juan Pablo Tosca Villanueva (May 16 2024 at 21:09):
Philip Durbin 🚀 (May 17 2024 at 02:08):Why is Windows so hard? :sob:
Juan Pablo Tosca Villanueva (May 17 2024 at 02:25)::upside_down:
Juan Pablo Tosca Villanueva (May 17 2024 at 14:07):So looking at the full log, here are some of the things that caught my eye:
|SEVERE|
ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console.
|INFO|
No deployment transformer implementation found.
|WARNING|
expected resource not found: /META-INF/javamail.default.address.map
|INFO|
PER01003: Deployment encountered SQL Exceptions:
PER01000: Got SQLException executing statement "ALTER TABLE fileaccessrequests ADD CONSTRAINT FK_fileaccessrequests_DATAFILE_ID FOREIGN KEY (DATAFILE_ID) REFERENCES DVOBJECT (ID)": org.postgresql.util.PSQLException: ERROR: constraint "fk_fileaccessrequests_datafile_id" for relation "fileaccessrequests" already exists
The "launch" (if you can call that a launch :laughing: ) takes about 10 minutes and because of that the race condition fails... now it seems that the application deploys and it is working per Payara info but it is EXTREMELY slow:
Juan Pablo Tosca Villanueva (May 17 2024 at 14:16):
Juan Pablo Tosca Villanueva (May 17 2024 at 14:17):I been several minutes calling "http://localhost:8080/" and the browser keeps loading lol but it won't show anything. It also took me like 5 minutes to get to payara console.
Juan Pablo Tosca Villanueva (May 17 2024 at 14:28):Ok so another update, I tried building the war and deploying it manually and it booms!
Juan Pablo Tosca Villanueva (May 17 2024 at 14:29):- Error in annotation processing: {0}.
2024-05-17 10:27:27 java.lang.NoClassDefFoundError: org/swordapp/server/CollectionListManager
Philip Durbin 🚀 (May 17 2024 at 15:03):@Victoria Lubitch where are you? Are you having same troubles?
Victoria Lubitch (May 17 2024 at 15:21):@Philip Durbin Yes, I think I have similar issues. I tried docker dataverse on another windows machine and it was so extremely slow, it was much more than 5 min, so I shut it off. On my other machine it was a bit faster but I do not get to dataverse app http://localhost:8080, only Payara Service page loads.
Philip Durbin 🚀 (May 17 2024 at 15:23):Thanks for joining! I wish I could help but without a Windows box I'm not sure what to do.
Juan Pablo Tosca Villanueva (May 17 2024 at 15:35):My windows machine is very powerful so it shouldn't be having issues... there is something wrong definitely but I just can't think of what platform specific differences that could be causing this... I am going back to rocky but I will try to give this a look later :upside_down:
Philip Durbin 🚀 (May 17 2024 at 15:40):Oh, now I understand the Rocky gif.
Philip Durbin 🚀 (May 17 2024 at 15:41):You're working on #10312. I get it.
Juan Pablo Tosca Villanueva (May 17 2024 at 15:41):Do you know why my branch is Rocky IX now?
Juan Pablo Tosca Villanueva (May 17 2024 at 15:41)::rolling_on_the_floor_laughing:
Philip Durbin 🚀 (May 17 2024 at 15:41):Let's move this to a rocky thread :grinning:
Juan Pablo Tosca Villanueva (May 17 2024 at 15:41):Nooooo
Juan Pablo Tosca Villanueva (May 17 2024 at 15:41):I am out, I will post with more windows updates :rolling_on_the_floor_laughing:
Philip Durbin 🚀 (May 17 2024 at 15:58):@Victoria Lubitch can you please share as much of the output as you can? Why the war file is not deploying, for example.
Victoria Lubitch (May 17 2024 at 16:39):It seems payara ha sproblem starting. I noticed if I get into localhost:4848 then it stuck into "The Admin console is starting. Please wait"
Victoria Lubitch (May 17 2024 at 16:48):dataverse_docker_win.log i attach the full log
Victoria Lubitch (May 17 2024 at 16:51):Ok, I finally got to payara console. What should be username and password for Administration console?
Victoria Lubitch (May 17 2024 at 16:55):localhost:8080 looks like in permanent process of loading, just circle.
Victoria Lubitch (May 17 2024 at 17:04):docker_win_more_errors.log
Now I got " jakarta.ejb.EJBException: getSingleResult() did not retrieve any entities."
Philip Durbin 🚀 (May 17 2024 at 17:21):username: admin
password: admin
should work for http://localhost:4848
Victoria Lubitch (May 17 2024 at 17:25):Thanks, dataverse is in the list of deployed applications
Finaly, I get "Page not found"
Screenshot-2024-05-17-132358.png
Philip Durbin 🚀 (May 17 2024 at 17:26):Ok, I assume http://localhost:8080/api/info/version is working.
Philip Durbin 🚀 (May 17 2024 at 17:26):Is http://localhost:8080/api/metadatablocks/citation working?
Victoria Lubitch (May 17 2024 at 17:26):It is all extremely slow.
Victoria Lubitch (May 17 2024 at 17:27):It is still loading
Philip Durbin 🚀 (May 17 2024 at 17:28):wow
Victoria Lubitch (May 17 2024 at 17:29):And my machine is not that bad, i have 32 G
Philip Durbin 🚀 (May 17 2024 at 17:30):Something is terribly wrong :scream:
Philip Durbin 🚀 (May 17 2024 at 17:30):In your logs I don't see "deployed" like I expect. I should go check what I see.
Juan Pablo Tosca Villanueva (May 17 2024 at 17:34):At 10:07 I posted results and logs from similar testing, same results. I also tried to make a war and deploy it and this was the error
Juan Pablo Tosca Villanueva (May 17 2024 at 17:34):Juan Pablo Tosca Villanueva said:
Victoria Lubitch (May 17 2024 at 17:36):Finally I got version {"status":"OK","data":{"version":"6.2","build":null}}
Victoria Lubitch (May 17 2024 at 17:36):But for citation {"status":"ERROR","message":"Can't find metadata block 'citation'"}
Philip Durbin 🚀 (May 17 2024 at 17:38):I'm not surprised "citation" isn't there. "info/version" is available as long as the war is deployed. But the setup scripts probably didn't run.
Juan Pablo Tosca Villanueva (May 17 2024 at 17:39):configbaker never runs
Philip Durbin 🚀 (May 17 2024 at 17:39):Here's what you should see: success.txt
Juan Pablo Tosca Villanueva (May 17 2024 at 17:39):If you use dockerdesktop probably could try to start the container
Philip Durbin 🚀 (May 17 2024 at 17:42):I look for this:
dev_dataverse> Loading application [dataverse] at [/]|#]
Followed by this:
dev_bootstrap> Now executing bootstrapping script at /scripts/bootstrap/dev/init.sh.
dev_bootstrap> Running base setup-all.sh (INSECURE MODE)...
But it sounds like ConfigBaker (dev_bootstrap) never runs?
Victoria Lubitch (May 17 2024 at 17:43):I have configbacker-1 and configbacker-2, It says "Excited". Should I try to run one of them?
Victoria Lubitch (May 17 2024 at 17:44):No, bootstrap init script does not look that it started to run.
Philip Durbin 🚀 (May 17 2024 at 17:44):I mean, I've very quick to kill and rm all containers and start over again. But it sounds like things are slow for you.
Philip Durbin 🚀 (May 17 2024 at 17:45):I just mean that I don't usually run configbaker separately, later. Usually, I just let it all run at once. But it might work.
Philip Durbin 🚀 (May 17 2024 at 17:46):JP, I'm not sure why you're seeing that weird error on deployment. At least Victoria is able to deploy. :sweat_smile:
Victoria Lubitch (May 17 2024 at 17:48):Extremely slow. I started configbaker-1 waiting for localhost:8080 to react.
Philip Durbin 🚀 (May 17 2024 at 17:51):It seems like the slowness is the biggest problem. Maybe configbaker timed out. :thinking:
Philip Durbin 🚀 (May 17 2024 at 17:51):Other containers run ok for you?
Victoria Lubitch (May 17 2024 at 17:51):It seems to take 50% of memory and 70% CPU
Victoria Lubitch (May 17 2024 at 17:52):Yes, other containers seems to be running.
Philip Durbin 🚀 (May 17 2024 at 17:52):Just for fun, could you please kill and rm all containers and try demo version instead? https://guides.dataverse.org/en/6.2/container/running/demo.html#quickstart
Victoria Lubitch (May 17 2024 at 17:55):Last time I run demo version it was ok. But I will try it again.
Philip Durbin 🚀 (May 17 2024 at 17:55):Oh! Interesting. That's good, I guess, that the demo works.
Philip Durbin 🚀 (May 17 2024 at 17:56):There's a lot of extra stuff developers don't need day to day in docker-compose-dev.yml: Keycloak, MinIO, LocalStack. Maybe we could remove some of that and see if it helps.
Juan Pablo Tosca Villanueva (May 17 2024 at 17:59):Bootstrap runs but it has a timer
Juan Pablo Tosca Villanueva (May 17 2024 at 17:59):so it checks for the version api
Juan Pablo Tosca Villanueva (May 17 2024 at 18:00):but as I mentioned, it takes more than 15 minutes on windows for DV to start
Philip Durbin 🚀 (May 17 2024 at 18:00):who has time for that?
Juan Pablo Tosca Villanueva (May 17 2024 at 18:00):The other error I posted was an additional test I made trying to do a manual deployment of the application since there are some weird things on the logs when it starts
Juan Pablo Tosca Villanueva (May 17 2024 at 18:01):Juan Pablo Tosca Villanueva said:
So looking at the full log, here are some of the things that caught my eye:
|SEVERE| ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console.|INFO| No deployment transformer implementation found.|WARNING| expected resource not found: /META-INF/javamail.default.address.map|INFO| PER01003: Deployment encountered SQL Exceptions: PER01000: Got SQLException executing statement "ALTER TABLE fileaccessrequests ADD CONSTRAINT FK_fileaccessrequests_DATAFILE_ID FOREIGN KEY (DATAFILE_ID) REFERENCES DVOBJECT (ID)": org.postgresql.util.PSQLException: ERROR: constraint "fk_fileaccessrequests_datafile_id" for relation "fileaccessrequests" already exists
Juan Pablo Tosca Villanueva (May 17 2024 at 18:01):That is what made me thing to try a manual deploy
Philip Durbin 🚀 (May 17 2024 at 18:06):Looking at your full log, you mean, right?
Victoria Lubitch (May 17 2024 at 18:11):Demo looks much better. I was able to login. Although I see some errors in log, some solr error and also again "jakarta.ejb.EJBException: getSingleResult() did not retrieve any entities."
Victoria Lubitch (May 17 2024 at 18:12):the solr error "2024-05-17 18:08:21.247 ERROR (qtp1997702454-27) [] o.a.s.s.HttpSolrCall 500 Exception => org.apache.solr.core.SolrCoreInitializationException: SolrCore 'collection1' is not available due to init failure: Could not load conf for core collection1: Can't load schema null: Error fetching schema"
Victoria Lubitch (May 17 2024 at 18:13):But UI seems functional.
Philip Durbin 🚀 (May 17 2024 at 18:17):Was it reasonably fast to spin up?
Victoria Lubitch (May 17 2024 at 18:20):Yes, solr definetly has some issues though. Should I be able create datasets with demo. I tried and after pressing Save dataset it turns like white screen frozen.
Philip Durbin 🚀 (May 17 2024 at 18:22):I might be related to Solr having trouble.
Philip Durbin 🚀 (May 17 2024 at 18:22):But could this demo env work for you? Or do you need a dev env?
Victoria Lubitch (May 17 2024 at 18:26):I need to eun develop branch, I was added to another project (Geoportal) that only runs on Windows and got new windows laptop. I still have old Ubuntu machine, but I thought If dataverse docker will work, I will be able to do occasional simple dev work on one machine instead of two :)
Philip Durbin 🚀 (May 17 2024 at 18:28):Ok, sounds like we should focus on the dev env, then. Should we try removing Keycloak, etc.
Victoria Lubitch (May 17 2024 at 18:30):Ok
Philip Durbin 🚀 (May 17 2024 at 18:32):Do you want to try? Do you need any guidance?
Philip Durbin 🚀 (May 17 2024 at 18:36):You could see if this is any better: docker-compose-dev.yml
I removed these:
Philip Durbin 🚀 (May 17 2024 at 18:37):... and it seems to work fine. I'm able to log in, etc.
Victoria Lubitch (May 17 2024 at 18:39):Ok, I will try that.
Philip Durbin 🚀 (May 17 2024 at 18:39):Great, I hope it's faster somehow. :shrug:
Victoria Lubitch (May 17 2024 at 19:33):So i took yours [docker-compose-dev.yml, renamed it as compose.yml and run $docker compose up. Does not to seem to work properly though. It deploys, I see it in payara console admin, but localhost:8080 still shows only payara page. Or maybe I am not doing things correctly?
Victoria Lubitch (May 17 2024 at 19:34):Configbaker running though this time.
Philip Durbin 🚀 (May 17 2024 at 19:37):Hmm, I don't THINK renaming the file should cause any problem.
Philip Durbin 🚀 (May 17 2024 at 19:38):Can you please send the output again? Is it faster?
Victoria Lubitch (May 17 2024 at 19:57):Yes, seems to be a little bit faster, although still slow.
dataverse_docker_win_dev.log
Philip Durbin 🚀 (May 17 2024 at 20:00):The log looks pretty good.
Philip Durbin 🚀 (May 17 2024 at 20:00):Did configbaker run?
Victoria Lubitch (May 17 2024 at 20:04):Yes
Philip Durbin 🚀 (May 17 2024 at 20:06):I don't see it in the log.
Philip Durbin 🚀 (May 17 2024 at 20:06):Not all the setup anyway. Creating the root collection, etc.
Victoria Lubitch (May 17 2024 at 20:20):After the long wait I get the same error "Page not found"
Philip Durbin 🚀 (May 17 2024 at 20:20):Is there more output?
Victoria Lubitch (May 17 2024 at 20:27):dataverse_docker_win_dev_more_errors.log
Philip Durbin 🚀 (May 17 2024 at 20:38):Huh. Nothing helpful there, unfortunately. Really I want to see what configbaker logged. Can you please run docker logs on it?
Victoria Lubitch (May 17 2024 at 20:45):Configbaker I see on docker images. dataverse container has dev_smtp, dev_bootstrap (not running) , dev_postgres, dev_solr_initializer (not running), dev_dv_initializer (not running), dev_solr, dev_dataverse and dataverse-dev-proxy1
Victoria Lubitch (May 17 2024 at 20:47):Which of their logs do you want to see?
Philip Durbin 🚀 (May 17 2024 at 20:49):Can you please try docker logs configbaker-1?
Victoria Lubitch (May 17 2024 at 20:52):there is no container configbaker-1, it is in images gdcc/configbaker
Philip Durbin 🚀 (May 17 2024 at 20:54):Can you click whatever it's called in Docker Desktop and see logs? Like this:
Screenshot-2024-05-17-at-4.53.52-PM.png
Victoria Lubitch (May 17 2024 at 20:58):dataverse_log.log
Screenshot-2024-05-17-165838.png
Philip Durbin 🚀 (May 17 2024 at 21:37):Ok, it looks like configbaker is called "dev_bootstrap". Can you please try docker logs dev_bootstrap?
Victoria Lubitch (May 21 2024 at 17:14):I get the following log
Screenshot-2024-05-21-131326.png
Victoria Lubitch (May 21 2024 at 17:17):i tried to install dataverse using wsl. It looked like it was working. I was able to login, create dataverse. But it fails on creating dataset. It said in the log that it cannot create persistent identifier " Command [DatasetCreate dataset:null] failed: Cannot invoke "edu.harvard.iq.dataverse.pidproviders.PidProvider.generatePid(edu.harvard.iq.dataverse.DvObject)" because "pidProvider" is null
edu.harvard.iq.dataverse.engine.command.exception.CommandException: Command [DatasetCreate dataset:null] failed: Cannot invoke "edu.harvard.iq.dataverse.pidproviders.PidProvider.generatePid(edu.harvard.iq.dataverse.DvObject)" because "pidProvider" is null
at edu.harvard.iq.dataverse.EjbDataverseEngine.submit(EjbDataverseEngine.java:287)" I use fake publishing.
Victoria Lubitch (May 21 2024 at 17:18):It looked that wsl is a more promissing way to have dataverse on windows, but I still did not figured out the dataset creation part.
Victoria Lubitch (May 21 2024 at 17:20):these are my pid jvm options:
-Ddataverse.pid.providers=fake
-Ddataverse.pid.fake.type=FAKE
-Ddataverse.pid.fake.label=Fake DOI Provider
-Ddataverse.pid.fake.authority=10.5072
-Ddataverse.pid.fake.shoulder=FK2/
Victoria Lubitch (May 21 2024 at 17:24):More logs for docker bootstap:
2024-05-21 13:22:14 Waiting for http://dataverse:8080 to become ready in max 3m.
2024-05-21 13:22:25 2024-05-21T17:22:25Z INF [HTTP] Checking the http://dataverse:8080/api/info/version ...
2024-05-21 13:22:28 2024-05-21T17:22:28Z ERR Expectation failed error="timed out while making an http call, caused by: Get \"http://dataverse:8080/api/info/version\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" timeout=3s
2024-05-21 13:22:36 2024-05-21T17:22:36Z INF [HTTP] Checking the http://dataverse:8080/api/info/version ...
2024-05-21 13:22:39 2024-05-21T17:22:39Z ERR Expectation failed error="timed out while making an http call, caused by: Get \"http://dataverse:8080/api/info/version\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" timeout=3s
2024-05-21 13:22:47 2024-05-21T17:22:47Z INF [HTTP] Checking the http://dataverse:8080/api/info/version ...
2024-05-21 13:22:50 2024-05-21T17:22:50Z ERR Expectation failed error="timed out while making an http call, caused by: Get \"http://dataverse:8080/api/info/version\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" timeout=3s
2024-05-21 13:22:58 2024-05-21T17:22:58Z INF [HTTP] Checking the http://dataverse:8080/api/info/version ...
2024-05-21 13:23:01 2024-05-21T17:23:01Z ERR Expectation failed error="timed out while making an http call, caused by: Get \"http://dataverse:8080/api/info/version\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" timeout=3s
2024-05-21 13:23:09 2024-05-21T17:23:09Z INF [HTTP] Checking the http://dataverse:8080/api/info/version ...
2024-05-21 13:23:12 2024-05-21T17:23:12Z ERR Expectation failed error="timed out while making an http call, caused by: Get \"http://dataverse:8080/api/info/version\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" timeout=3s
2024-05-21 13:23:20 2024-05-21T17:23:20Z INF [HTTP] Checking the http://dataverse:8080/api/info/version ...
2024-05-21 13:23:23 2024-05-21T17:23:23Z ERR Expectation failed error="timed out while making an http call, caused by: Get \"http://dataverse:8080/api/info/version\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" timeout=3s
Philip Durbin 🚀 (May 21 2024 at 17:42):Oh! You're so close! Can you please add this one?
-Ddataverse.pid.default-provider=fake
Philip Durbin 🚀 (May 21 2024 at 17:44):It should be there already...
$ grep default-provider docker-compose-dev.yml -B1 -A4
-Ddataverse.pid.providers=fake
-Ddataverse.pid.default-provider=fake
-Ddataverse.pid.fake.type=FAKE
-Ddataverse.pid.fake.label=FakeDOIProvider
-Ddataverse.pid.fake.authority=10.5072
-Ddataverse.pid.fake.shoulder=FK2/
yes, it worked! I was able to create a dataset. So wsl is working, that is good enough for me. Thanks Philip for help. Just need to check how it works with S3.
Philip Durbin 🚀 (May 21 2024 at 18:56):@Victoria Lubitch fantastic! Please, please can you make a pull request for the container guide to add something helpful for the next Windows user?
Victoria Lubitch (May 21 2024 at 19:06):Ok, I will add guide for using WSL in Windows for dataverse installation here https://guides.dataverse.org/en/latest/developers/windows.html It should be enough for development on windows machine.
Although Docker in windows does not work yet unfortunately.
Philip Durbin 🚀 (May 21 2024 at 19:41):Sounds great! Thanks!
Philip Durbin 🚀 (Aug 20 2024 at 19:57):@Bethany Seeger heads up that @Victoria Lubitch recently added some instructions for setting up a dev environment on Windows. For now you can preview them at https://preview.guides.gdcc.io/en/develop/developers/windows.html
Philip Durbin 🚀 (Aug 20 2024 at 19:58):The PR is #10608 and it will be part of the 6.4 guides. The above is just a preview.
Bethany Seeger (Aug 20 2024 at 20:52):Great! Thanks, @Philip Durbin !
Philip Durbin 🚀 (Jun 17 2025 at 15:50):@Victoria Lubitch thanks for sitting with me during the DCM25 hackathon.
As we discussed, I just made this PR:
use WSL not cmd.exe for Windows dev, simplify guides #11583
Philip Durbin 🚀 (Jun 17 2025 at 15:50):Please take a look and let me know if you want any changes.
Philip Durbin 🚀 (Jun 17 2025 at 15:51):@Bethany Seeger I forget, were you looking for help getting a dev environment on Windows as well? Nice meeting you last week by the way!
Bethany Seeger (Jul 23 2025 at 20:30):Hi @Philip Durbin 🚀 I had been thinking about it, but not for quite a while. I have some good dev and test space in our servers, so I haven't had a huge need for it. But, I will read though this PR to learn more.
It was nice to meet you as well! :)
Philip Durbin 🚀 (Jul 23 2025 at 20:31):Awesome. That PR was tested by @Victoria Lubitch @Juan Pablo Tosca Villanueva and Jim so I'm feeling pretty good about it (even though I don't have Windows and can't test it myself). :smile:
Philip Durbin 🚀 (Jul 23 2025 at 20:32):And yes, always nice to put a name with a face. :smile:
Last updated: Oct 30 2025 at 05:14 UTC