Philip Durbin 🚀 (Sep 11 2023 at 02:36): Philip Durbin 🚀 (Sep 11 2023 at 02:36):This report just came in: https://groups.google.com/g/dataverse-community/c/Bpbj2b3V74w/m/WljyMJlpAgAJ
Philip Durbin 🚀 (Sep 11 2023 at 11:12):@Oliver Bertuch thanks for replying and encouraging Juan Pablo to join us here!
Philip Durbin 🚀 (Sep 11 2023 at 11:13):It sounds like he might even be able to bring his laptop by my office, if necessary: https://github.com/IQSS/dataverse/pull/9895#issuecomment-1712271029
Oliver Bertuch (Sep 11 2023 at 11:29):Anytime!
Philip Durbin 🚀 (Sep 11 2023 at 13:17):@Akio Sone you're on Windows. Any thoughts on this?
Philip Durbin 🚀 (Sep 11 2023 at 13:17):@Oliver Bertuch would it be crazy to have https://github.com/gdcc/api-test-runner test Windows?
Juan Pablo Tosca Villanueva (Sep 11 2023 at 14:31):Hi everyone! Thanks for your help @Oliver Bertuch I just posted more information and If I get any findings today, I will post them.
Philip Durbin 🚀 (Sep 11 2023 at 14:40):So if you chmod +x the first script, it executes? But there are a lot of scripts! :sweat_smile:
Philip Durbin 🚀 (Sep 11 2023 at 14:41):If you want to try rebuilding the configbaker images, please see #9885.
Philip Durbin 🚀 (Sep 11 2023 at 14:42):Just as a workaround until we figure this out.
Juan Pablo Tosca Villanueva (Sep 11 2023 at 14:42):I will try that briefly and let you know if I have any luck @Philip Durbin ! thanks :smile:
Philip Durbin 🚀 (Sep 11 2023 at 14:43):I can't remember if @Akio Sone tested configbaker or just ran the equivalent scripts (back before we had configbaker) on Windows.
Juan Pablo Tosca Villanueva (Sep 11 2023 at 14:52):Hi @Philip Durbin I am reading through #9885 but when I click on the link "https://dataverse.zulipchat.com/#narrow/stream/375812-containers/topic/iterating.20on.20configbaker" It ends in a "no search results" page, Is this thread behind some credentials or expired maybe?
Philip Durbin 🚀 (Sep 11 2023 at 14:53):D'oh! I was afraid of that. Please try #containers > ✔ iterating on configbaker
Juan Pablo Tosca Villanueva (Sep 11 2023 at 14:54):Thanks!
Oliver Bertuch (Sep 11 2023 at 16:14):Hi @Juan Pablo Tosca Villanueva !
Oliver Bertuch (Sep 11 2023 at 16:14):Welcome!
Oliver Bertuch (Sep 11 2023 at 16:14):You asked about how to get deets for B) and C)
B) Can you verify that the scripts have the executable bit set inside the container?
C) Can you verify that the container is running as root? (It shouldn't matter, but lets see if there might be a problem)
Oliver Bertuch (Sep 11 2023 at 16:15):If you run docker run -it --rm gdcc/configbaker:unstable bash, you can explore the filesystem.
Oliver Bertuch (Sep 11 2023 at 16:17):You can look around and explore /scripts. When analyzing the content with dive, I see the following:
image.png
Oliver Bertuch (Sep 11 2023 at 16:17):So all the permission bits set. Let's see if those prevail on a windows container host
Oliver Bertuch (Sep 11 2023 at 16:18):(BTW I probably will have a Win CT host at my fingertips as of tomorrow and can test a few things, too)
Juan Pablo Tosca Villanueva (Sep 11 2023 at 20:42):I am going to try this later tonight and let you know my findings. :smile:
Juan Pablo Tosca Villanueva (Sep 11 2023 at 23:13):Let's see, I am not sure I am looking at the right thing, just downloaded dive for the first time and checked the gdcc/configbaker image and this is the content:
image.png
Philip Durbin 🚀 (Sep 11 2023 at 23:59):Yeah, that looks right. A few scripts.
Philip Durbin 🚀 (Sep 12 2023 at 00:00):If it helps, here's how it looks as of 6.0: https://github.com/IQSS/dataverse/tree/v6.0/modules/container-configbaker
Philip Durbin 🚀 (Sep 12 2023 at 00:00):Docs: https://guides.dataverse.org/en/6.0/container/configbaker-image.html
Philip Durbin 🚀 (Sep 12 2023 at 00:01):Please keep the questions coming! :sweat_smile:
Oliver Bertuch (Sep 12 2023 at 00:32):Please note that after the copy, there is a run statement that should reveal the chmod +x I'm running on purpose there.
Oliver Bertuch (Sep 12 2023 at 00:32):Would be good to check from a running container if it actually comes through though.
Oliver Bertuch (Sep 12 2023 at 00:32):It does on my Linux box... :see_no_evil:
Juan Pablo Tosca Villanueva (Sep 12 2023 at 17:44):It seems that I got it working or at least I made some progress, just a word of warning. If you modify a script with the docker UI on the container it will change the permissions, so the files are not able to run.
Juan Pablo Tosca Villanueva (Sep 12 2023 at 17:46):Ok it is working:
image.png
Philip Durbin 🚀 (Sep 12 2023 at 17:46):Nice! Would you like to make a pull request?
Juan Pablo Tosca Villanueva (Sep 12 2023 at 18:00):Sure! let me just share what the change is and let me know your thoughts:
First after creating the docker image from a fresh installation, I checked the files with dive and all the scripts seem to have the right permissions:
But there is another set of scripts that seems to me that are created in another point during the launch of the container, these files are the following:
For what I see these are created by the dockerfile
What I did to fix the permissions is add the following line to /dev/init.sh
chmod +x "${BOOTSTRAP_DIR}"/base/*.sh
this fixed the permissions and let the execution to complete, now I think this may not be the right solution so I may do some testing today.
For what I see on /modules/container-configbaker/dockerfile at line 31 there is a RUN chmod to set the permissions on ${SCRIPT_DIR}/.sh and ${BOOTSTRAP_DIR}//*.sh but is not until line 40 where these files are copied so they are left out of this.
I am going to test this in a few and let you know.
Philip Durbin 🚀 (Sep 12 2023 at 18:05):Right, those scripts get included based on this config file from what I understand: https://github.com/IQSS/dataverse/blob/v6.0/modules/container-configbaker/assembly.xml#L19-L22
Philip Durbin 🚀 (Sep 12 2023 at 18:05):It's just a list. I don't know if we can specify in that XML file that certain files should be executable.
Juan Pablo Tosca Villanueva (Sep 12 2023 at 18:07):In the dockerfile there is this code:
image.png
the red is when it set the permissions but is not up to blue when these files are copied
Philip Durbin 🚀 (Sep 12 2023 at 18:09):Gotcha.
Philip Durbin 🚀 (Sep 12 2023 at 18:09):Here I'm seeing something about Windows:
"keep to respect the assembly provided permissions, exec for setting the executable bit on all files (required for Windows when using an assembly mode dir)"
https://dmp.fabric8.io/#build-assembly
Philip Durbin 🚀 (Sep 12 2023 at 18:11):There's also fileMode at https://maven.apache.org/plugins/maven-assembly-plugin/assembly.html and the example they give is 0644.
Philip Durbin 🚀 (Sep 12 2023 at 18:11):@Juan Pablo Tosca Villanueva if your chmod is working, that's fine. A PR would be appreciated!
Philip Durbin 🚀 (Sep 12 2023 at 18:16):And an issue, please, like you did last time. :big_smile:
Juan Pablo Tosca Villanueva (Sep 12 2023 at 18:17):Would this be a Bug report or a Feature request :sweat_smile: (It may not be a bug it is a feature? :rolling_on_the_floor_laughing: )
Philip Durbin 🚀 (Sep 12 2023 at 18:19):It's a bug. "Dockerized Dataverse doesn't work on Windows" or whatever.
Philip Durbin 🚀 (Sep 12 2023 at 18:35):#9904 looks great. Thanks!
Juan Pablo Tosca Villanueva (Sep 12 2023 at 18:45):Thanks @Philip Durbin
Could you offer me some guidance with the steps to create another PR? It seems that the previous one https://github.com/IQSS/dataverse/pull/9895 is having an issue with "continuous-integration/jenkins/pr-merge — This commit cannot be built", when I click details, the page can't load.
image.png
Since I can't create a second fork should I delete this one ?
Philip Durbin 🚀 (Sep 12 2023 at 18:58):Two, things:
Our CI is broken right now, sorry!
Yes, I noticed you used the develop branch on your fork last time. Maybe you could close that PR, delete the fork, fork again, and create branches next time. Here's how we describe this: https://guides.dataverse.org/en/6.0/developers/version-control.html#how-to-make-a-pull-request
Sometimes a develop branch on a fork gives us trouble: #9508.
Juan Pablo Tosca Villanueva (Sep 12 2023 at 19:04):Ok going to grave something across the street and will look at this! Thanks.
Juan Pablo Tosca Villanueva (Sep 12 2023 at 19:41):Hi @Philip Durbin I am not sure, but do I need to request access to create a branch? It seems that is the case because I get a message like this:
Philip Durbin 🚀 (Sep 12 2023 at 20:04):Ah, are you using the GitHub UI, primarily? It should all be possible. You'll want to click that "create a fork" button, assuming you deleted your old fork.
I'm happy to jump on Zoom if you like!
Juan Pablo Tosca Villanueva (Sep 12 2023 at 20:14):I normally use the CLI or the web interface but was testing this to avoid the same issue. It would be great if I can get you on zoom, let me know when is the best time for you :smile:
Philip Durbin 🚀 (Sep 12 2023 at 20:22):https://harvard.zoom.us/my/pdurbin?pwd=em1WNUZGbnY2YjhxNEdSbjJJMXNSUT09
Philip Durbin 🚀 (Sep 12 2023 at 20:22):(If you have time now. If not, no worries!)
Juan Pablo Tosca Villanueva (Sep 12 2023 at 21:11):The change on the "dockerfile" definitely works and seems a better solution. I will do a PR tonight after a little bit more of testing.
Philip Durbin 🚀 (Sep 12 2023 at 21:11):Much appreciated!
Juan Pablo Tosca Villanueva (Sep 12 2023 at 21:43):Thanks to you for walking me through the GIT process and to @Oliver Bertuch for sharing dive and the bash command for docker, really useful. Let me know if you guys want to test on a windows machine it is just a bus (73) to get to the square. :smile:
Oliver Bertuch (Sep 13 2023 at 05:48):So I was able to install Docker Desktop on a brand new Windows 11 machine yesterday. Using WSL2 instead of HyperV. Pulling the images and exploring around, I cannot see that the executable bit would be missing on any of these scripts. Now I'm wondering which way to setup Linux containers on Windows you choose, which version of the underlying distribution you're using etc.
Oliver Bertuch (Sep 13 2023 at 06:36):OK I was able to kind of reproduce the problem.
Oliver Bertuch (Sep 13 2023 at 06:37):Using IntelliJ on Windows, cloning the project to the windows host (NOT the WSL machine!), I tried to build the image from scratch
Oliver Bertuch (Sep 13 2023 at 06:38):And, as you said, all of a sudden the executable bit was missing
Oliver Bertuch (Sep 13 2023 at 06:38):However, when I pulled the images from Docker Hub, they were not missing.
Oliver Bertuch (Sep 13 2023 at 06:43):@Juan Pablo Tosca Villanueva if you run docker in a powershell (NOT in the WSL subsystem), can you remove the configbaker image and pull fresh from Docker Hub? (The image id should be c21429de3745) Using this command should directly reveal the details:
image.png
Oliver Bertuch (Sep 13 2023 at 06:48):Rebuilding the image as described on the Windows host (files were cloned to Windoze file system), the image ID changed:
image.png
Running the same command again, I now see the missing exec bits:
image.png
Oliver Bertuch (Sep 13 2023 at 06:50):@Juan Pablo Tosca Villanueva did you rebuild the images before using? Because when "just" pulling from the Hub, it should have worked. The reason the hub images have the exec bit is because CI runs on Linux, keeping the git tracked exec bit intact. Checking out on a Win host cannot keep that intact (so your PR is indeed correct - we should take care of that for people! Thank you! :heart: )
Oliver Bertuch (Sep 13 2023 at 06:51):But still I would like to evaluate if it works from "just pulling" - otherwise we'd have a problem at our hands!
Oliver Bertuch (Sep 13 2023 at 06:59):Oh and we could try to checkout a project in WSL instead of on the Windows host. IntelliJ is fine with that, should work with VSCode as well. Dunno if that would be a weird workflow, if there is such an easy fix.
Juan Pablo Tosca Villanueva (Sep 13 2023 at 13:50):So basically I tried to follow the documentation to the best of my abilities, so I started on the windows page and this is where I proposed the change to the .gitattributes, downloaded and installed jq and as mentioned on the documentation moved over Development Usage.
Jumped right into the building section and ran "mvn -Pct clean package" and this is where the odyssey started :rolling_on_the_floor_laughing:. Is there a pre-built version of the configbaker container? It was under my impression that this was created on the target.
Philip Durbin 🚀 (Sep 13 2023 at 13:59):The path you took makes perfect sense to me. Yes, there is a pre-built image on Docker Hub but a new developer should run clean package like you did. In the quickstart we combine in into one command: mvn -Pct clean package docker:run
Philip Durbin 🚀 (Sep 13 2023 at 14:00):https://guides.dataverse.org/en/6.0/developers/dev-environment.html#quickstart
Oliver Bertuch (Sep 13 2023 at 14:03):and ran "mvn -Pct clean package"
Aha! So that explains it - you built the images, images on the hub are fine. Phew! :sweat_smile:
Oliver Bertuch (Sep 13 2023 at 14:06):WRT the PR: we probably should just merge the chmod actions into one at the end. Seems cleaner, less layers.
Oliver Bertuch (Sep 13 2023 at 14:08):(Or move copying the bootstrap script to the top, before the first chmod)
Oliver Bertuch (Sep 13 2023 at 14:08):Yeah, looking the configbaker dockerfile, that's probably the best option
Juan Pablo Tosca Villanueva (Sep 13 2023 at 14:09):That was my original idea but the only thing that stopped me from doing that it was the RUN on the managed-schema.xml I am not sure this need any permissions at this point
image.png
Juan Pablo Tosca Villanueva (Sep 13 2023 at 14:10):So I didn't wanted to break something else on the way :rolling_on_the_floor_laughing:
Oliver Bertuch (Sep 13 2023 at 14:11):No, that is completely unrelated
Juan Pablo Tosca Villanueva (Sep 13 2023 at 14:14):Then yes, I think running that at the end would make total sense :smile:
Juan Pablo Tosca Villanueva (Sep 13 2023 at 16:26):I will make the changes to put it at the end later today and update the PR @Philip Durbin @Oliver Bertuch I had some workshops today but I am done with the last one after 3:00 est :smile:
Oliver Bertuch (Sep 13 2023 at 16:27):I will try to give this some more thought - it is also touching on layer optimization. Might update the PR.
Oliver Bertuch (Sep 13 2023 at 16:27):Thanks for working on this!
Juan Pablo Tosca Villanueva (Sep 13 2023 at 16:31):Sure, just let me know. Thanks to both of you for all the help and being so responsive! :tada:
Juan Pablo Tosca Villanueva (Sep 22 2023 at 14:34):Oliver Bertuch said:
Thanks for working on this!
Hi Oliver, good morning. You mentioned you wanted to give this some more thought. What do you think should be the best solution here. Should I move the permissions to the end and not having this logic duplicated?
Philip Durbin 🚀 (Sep 22 2023 at 14:44):I believe he listened to yesterday's meeting (thanks for coming) and didn't object to how I summarized things, so I think you should try moving that copy up (and test it, of course). I can leave a ```suggestion in the PR if you'd like.
Juan Pablo Tosca Villanueva (Sep 22 2023 at 16:25):Ok It is up! :D Let me know your thoughts.
Philip Durbin 🚀 (Sep 22 2023 at 17:33):/me looks at #9906
Juan Pablo Tosca Villanueva (Sep 22 2023 at 17:33):I tested this on a fresh installation and it has the same effect than the previous change, the configbaker-1 will run successfully.
Previous comment moved to https://dataverse.zulipchat.com/#narrow/stream/375812-containers/topic/Dataverse-1.20container.20missing.20deployment.20on.20first.20launch.2E
Philip Durbin 🚀 (Sep 22 2023 at 17:35):#9906 looks good to me but Oliver's the one who wanted the copy moved so I'll let him weigh in. Thanks!
Philip Durbin 🚀 (Sep 22 2023 at 17:36):Hmm, you shouldn't need to restart containers... new topic, please! :big_smile: Or you can move that message to a new topic if that's easier.
Juan Pablo Tosca Villanueva (Sep 22 2023 at 17:37):I will open a new issue for the sake of clarity.
Philip Durbin 🚀 (Sep 22 2023 at 17:37):I love clarity!
Philip Durbin 🚀 (Sep 22 2023 at 18:31):I kicked off a test run here: https://github.com/gdcc/api-test-runner/actions/runs/6277848900
Philip Durbin 🚀 (Sep 22 2023 at 18:31):But I haven't used that runner in a few days (and our API tests can be flaky) so let's not freak out if there are failures. :sweat_smile:
Juan Pablo Tosca Villanueva (Sep 22 2023 at 18:41):Ok #9949 is open :sweat_smile:
Philip Durbin 🚀 (Sep 22 2023 at 18:44):Ok, but a new topic here in Zulip if we want to discuss. Thanks!
Last updated: Oct 30 2025 at 05:14 UTC