Jan Range (May 03 2023 at 13:58): Jan Range (May 03 2023 at 13:58):@Philip Durbin @Oliver Bertuch do you have experience in sharing an action to the marketplace? Those that I did always included a script or something. Is it possible to just share the yml in its current form?
Philip Durbin 🚀 (May 03 2023 at 13:59):@Jan Range no but @Ana Trisovic does!
Philip Durbin 🚀 (May 03 2023 at 13:59):Check out https://github.com/marketplace/actions/dataverse-uploader-action !
Jan Range (May 03 2023 at 14:01):Oh nice! Thanks for sending
Oliver Bertuch (May 03 2023 at 14:36):What would you like to achieve with that action?
Jan Range (May 03 2023 at 14:41):I would like to bundle all of this, so that end users can simply call this as a step. Nothing fancy, just to reduce boilterplate.
steps:
- name: Setup Dataverse
uses: IQSS/dataverse-dev@master
Well we can create a composite action for this https://docs.github.com/en/actions/creating-actions/creating-a-composite-action
Oliver Bertuch (May 03 2023 at 14:43):But we would need to think about stuff like variables etc
Oliver Bertuch (May 03 2023 at 14:44):Certainly those would be inputs...
Oliver Bertuch (May 03 2023 at 14:44):And I'm not sure if the compose thing is the best way to go here
Oliver Bertuch (May 03 2023 at 14:45):It's hard to put in env vars for the container
Oliver Bertuch (May 03 2023 at 14:48):Ah wait, if we go for an action, we might create a compose file that references an env_file in the flesh... That might work for generating these vars before we start the container
Oliver Bertuch (May 03 2023 at 14:50):OK so we need as inputs:
Anything else?
Oliver Bertuch (May 03 2023 at 15:25):@Guillermo Portas would such an action be of use for you as well?
Guillermo Portas (May 04 2023 at 17:59):@Oliver Bertuch Of course! That action would be useful for running frontend e2e and integration tests in an automated way through GitHub actions
Philip Durbin 🚀 (May 09 2023 at 13:26):@Oliver Bertuch is configbaker part of this?
Oliver Bertuch (May 09 2023 at 13:39):Absolutely
Oliver Bertuch (May 09 2023 at 13:39):That's what will provide the hard part of bootstrapping et al
Philip Durbin 🚀 (May 09 2023 at 13:46):Ok, so it's a dependency, basically. Thanks.
Philip Durbin 🚀 (Aug 30 2023 at 14:42):@Jan Range and I talked about this briefly today. Stay tuned! :big_smile:
Philip Durbin 🚀 (Sep 07 2023 at 10:46):We ended up creating https://github.com/gdcc/dataverse-action (a private repo) but no action there yet. :big_smile:
Jan Range (Sep 18 2023 at 05:42):Philip Durbin schrieb:
We ended up creating https://github.com/gdcc/dataverse-action (a private repo) but no action there yet. :big_smile:
Hi Phil! Sorry for the delay, I was on vacation until today. Going to push the action today :-)
Jan Range (Sep 18 2023 at 10:24):I just pushed the action to the repository. Here is an example run
https://github.com/gdcc/dataverse-action/actions/runs/6221210931/job/16882733005
Jan Range (Sep 18 2023 at 10:27):I am running into issues when creating a dataverse:
{"status":"ERROR","message":"Can't find dataverse with identifier='1'"}
Tried it also with root, but no chance. My guess is that the respective dataverse hasnt been created yet - Container setup might lagging. If that is the cause, is there an API endpoint I can use to check?
It worked once though, but at that point the builtin user I have created does not have permission yet. How can I set this up? The given API_TOKEN should enable testers to fully operate without permission issues.
Oliver Bertuch (Sep 18 2023 at 10:49):@Jan Range should we talk?
Jan Range (Sep 18 2023 at 10:50):I was able to resolve the issue by adding a health check for the dataverse URL ".../api/dataverses/root", but now there is a new error message:
🚀 Creating Test Dataverse
147
{"status":"ERROR","message":"Command edu.harvard.iq.dataverse.engine.command.impl.CreateDataverseCommand@5424d2a6 failed: Exception thrown from bean: jakarta.ejb.EJBTransactionRolledbackException: Exception thrown from bean: org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException: Error from server at http://solr:8983/solr/collection1: SolrCore 'collection1' is not available due to init failure: Could not load conf for core collection1: Can't load schema /var/solr/data/collection1/conf/schema.xml: Plugin init failure for [schema.xml] fieldType \"text_ws\": Plugin init failure for [schema.xml] analyzer/tokenizer \"whitespace\": [schema.xml] analyzer/tokenizer: missing mandatory attribute 'class'"}
We should tweak a few things probably...
Jan Range (Sep 18 2023 at 10:51):@Oliver Bertuch sounds great! I have a meeting now and will be available again around 14.00. Would that work for you?
Oliver Bertuch (Sep 18 2023 at 10:52):Sure, should work
Jan Range (Sep 18 2023 at 10:55):Alright, does WebEx work for you? Otherwise we can also use Zoom if you have a free room or sth
Oliver Bertuch (Sep 18 2023 at 10:55):I can provide a Zoom room
Jan Range (Sep 18 2023 at 10:55)::+1:
Oliver Bertuch (Sep 18 2023 at 11:53):https://fz-juelich-de.zoom.us/j/69281413267?pwd=bDZGbFVGM1A3U0lkT3JjOE9WdkpNQT09
@Jan Range @Philip Durbin
Jan Range (Sep 18 2023 at 11:55):Joining in a minute :-)
Oliver Bertuch (Sep 18 2023 at 11:55):See ya on the other side
Oliver Bertuch (Sep 18 2023 at 13:25):So we worked on the action a little and created two issues: https://github.com/gdcc/dataverse-action/issues/2 and https://github.com/gdcc/dataverse-action/issues/1
Oliver Bertuch (Sep 18 2023 at 13:25):We want functionality from upstream! :smiley:
Notification Bot (Sep 18 2023 at 13:51):2 messages were moved here from #python > downloading a file by dataset DOI and filename by Philip Durbin.
Philip Durbin 🚀 (Sep 18 2023 at 16:08):Interesting issues. :big_smile:
Oliver Bertuch (Sep 18 2023 at 17:53):Say more...? :thinking:
Philip Durbin 🚀 (Sep 18 2023 at 17:58):I'm just looking forward to hearing more about it. I just put it on the agenda for Thursday: https://ct.gdcc.io :big_smile:
Oliver Bertuch (Sep 18 2023 at 18:54):Here's an idea for shipping the postgres/solr version: how about we use labels/annotations?
Philip Durbin 🚀 (Sep 18 2023 at 18:56):No objection.
Oliver Bertuch (Sep 18 2023 at 18:56):We could use sth like "org.dataverse.deps.postgresql=13" and "org.dataverse.deps.solr=9.3.0"
Oliver Bertuch (Sep 18 2023 at 18:57):These can be added when building the image, Maven will provide those
Philip Durbin 🚀 (Sep 18 2023 at 18:58):Sure, sounds fancy. :monocle:
Oliver Bertuch (Sep 18 2023 at 18:58):This information definitely is metadata about the application, so it should be treated as metadata
Oliver Bertuch (Sep 18 2023 at 18:59):That's what annotations are supposed to be used for
Jan Range (Sep 19 2023 at 09:55):@Oliver Bertuch since we've added the bootstrap to the action, will the API Token be present in the environment? I remember that this will be written to it if I am not mistaking. If so, we could add a small step moving it into the Action Env.
Oliver Bertuch (Sep 19 2023 at 11:32):I definitely wanted to make the API token available as an output of the action
Oliver Bertuch (Sep 19 2023 at 11:32):But I don't think it'll be around yet
Oliver Bertuch (Sep 19 2023 at 11:33):Need to look at the scripts again - that token is extracted at some place
Oliver Bertuch (Sep 19 2023 at 11:33):So we should be able to expose it
Jan Range (Sep 19 2023 at 11:45):Just checked the scripts and it is actually exported. Hence, should be available at runtime.
We could write it to the GitHub env by using this line right after bootstrapping:
echo "API_TOKEN=$API_TOKEN" >>$GITHUB_ENV
That way it is accessible in successive steps within the action. Shall I add and test it?
Oliver Bertuch (Sep 19 2023 at 11:46):We could, but that is only within the action!
Jan Range (Sep 19 2023 at 11:46):Alright, so you were thinking of having this as a general output of the bootstrapping alongside other outputs? Makes sense
Oliver Bertuch (Sep 19 2023 at 11:47):Wait - actually what you are proposing... I doubt it'll work
Oliver Bertuch (Sep 19 2023 at 11:47):The token is exposed within the script running in a container
Oliver Bertuch (Sep 19 2023 at 11:47):That means the outer layer of our action will not know about it
Jan Range (Sep 19 2023 at 11:48):Oh okay, that makes sense. So there needs to be a way to extract it.
Jan Range (Sep 19 2023 at 11:49):Stackoverflow ftw :-D
https://stackoverflow.com/questions/34051747/get-environment-variable-from-docker-container
Oliver Bertuch (Sep 19 2023 at 11:50):No sir, that is not going to help us... The bootstrap container will not be running any longer after it's done
Jan Range (Sep 19 2023 at 11:51):Hm thats unfortunate :-D
Philip Durbin 🚀 (Sep 19 2023 at 11:51):Do you need the dataverseAdmin API token? To run tests or whatever, can you just create a new user? If the admin API isn't blocked that user can even make themselves a superuser.
Oliver Bertuch (Sep 19 2023 at 11:51):So we could mount some dir to drop a file into
Oliver Bertuch (Sep 19 2023 at 11:52):Let me evaluate one other simple idea
Oliver Bertuch (Sep 19 2023 at 11:52):We could use another output channel
Oliver Bertuch (Sep 19 2023 at 11:52):We have fd 0, 1 and 2 by default. Let me see if I can make use of 3 to output something
Oliver Bertuch (Sep 19 2023 at 11:53):output redirection for the win
Jan Range (Sep 19 2023 at 11:55):@Philip Durbin yes, we were thinking of the bootstrap process to provide the API Token right after finishing. Done it the way you described too, but havent tried to permit super admin rights. How can that be done?
Oliver Bertuch (Sep 19 2023 at 12:00):Hmm no we need to mount a file or directory
Philip Durbin 🚀 (Sep 19 2023 at 12:01):https://guides.dataverse.org/en/6.0/api/native-api.html#make-user-a-superuser
Oliver Bertuch (Sep 19 2023 at 12:02):OK this needs upstream support...
Oliver Bertuch (Sep 19 2023 at 12:02):I'm going to create a new issue in upstream
Oliver Bertuch (Sep 19 2023 at 14:26):So I added https://github.com/IQSS/dataverse/issues/9933 and created https://github.com/IQSS/dataverse/pull/9935
Oliver Bertuch (Sep 19 2023 at 14:26):Have a look :-)
Oliver Bertuch (Sep 19 2023 at 14:26):We probably need to get some stuff merged to continue with the action...
Philip Durbin 🚀 (Sep 19 2023 at 14:58):"Modify compose file to mount an existing file into the bootstrap container"... should we go ahead and add this to the dev compose file? Maybe commented out?
Oliver Bertuch (Sep 19 2023 at 15:17):Hmm. This use case might be far stretched from what we do in the main repo. Currently, I can't think of a usage there... But I might be wrong.
Philip Durbin 🚀 (Sep 19 2023 at 15:25):Ok. In the PR can you show a diff of what you mean?
Oliver Bertuch (Sep 19 2023 at 16:15):You mean an example in the description?
Philip Durbin 🚀 (Sep 19 2023 at 18:05):Like this?
diff --git a/docker-compose-dev.yml b/docker-compose-dev.yml
index ab44dbc180..d6129cf738 100644
--- a/docker-compose-dev.yml
+++ b/docker-compose-dev.yml
@@ -41,6 +41,8 @@ services:
command:
- bootstrap.sh
- dev
+ - -e
+ - /tmp/envfile.txt
networks:
- dataverse
Close. Order is wrong. And it needs a volume mount for the file.
Philip Durbin 🚀 (Sep 19 2023 at 18:10):This why I'm asking for a diff :sweat_smile:
Oliver Bertuch (Sep 19 2023 at 18:10):Hrhr
Oliver Bertuch (Sep 19 2023 at 18:12): dev_bootstrap:
container_name: "dev_bootstrap"
image: gdcc/configbaker:unstable
restart: "no"
command:
- bootstrap.sh
+ - -e
+ - /tmp/envfile.txt
- dev
+ volumes:
+ - ./envfile.txt:/tmp/envfile.txt
networks:
- dataverse
But it is important to create that file on the host first - otherwise you will end up with a folder called "envfile.txt"... :wink:
Philip Durbin 🚀 (Sep 19 2023 at 18:23):Uh oh:
$ file /tmp/envfile.txt
/tmp/envfile.txt: empty
oh dev needs to be last
Philip Durbin 🚀 (Sep 19 2023 at 18:29):closer, I think, but...
dev_bootstrap> File /tmp/envfile.txt not found, is a directory or not writeable
Oliver Bertuch (Sep 19 2023 at 18:29):Did you mount the file?
Philip Durbin 🚀 (Sep 19 2023 at 18:29):$ file /tmp/envfile.txt
/tmp/envfile.txt: empty
$ cd /tmp/envfile.txt
-bash: cd: /tmp/envfile.txt: Not a directory
$ ls -l /tmp/envfile.txt
-rw-r--r-- 1 pdurbin wheel 0 Sep 19 14:05 /tmp/envfile.txt
diff --git a/docker-compose-dev.yml b/docker-compose-dev.yml
index ab44dbc180..82f320c1e9 100644
--- a/docker-compose-dev.yml
+++ b/docker-compose-dev.yml
@@ -40,7 +40,11 @@ services:
restart: "no"
command:
- bootstrap.sh
+ - -e
+ - /tmp/envfile.txt
- dev
+ volumes:
+ - ./envfile.txt:/tmp/envfile.txt
networks:
- dataverse
I'm glad we're talking about this here and not in the PR. :sweat_smile: Chat is so nice.
Oliver Bertuch (Sep 19 2023 at 18:37):The container should be running as root, right? So it should have the required permissions...
Philip Durbin 🚀 (Sep 19 2023 at 18:43):/Library/PrivilegedHelperTools/com.docker.vmnetd is running as root but all other docker things from ps aux are running as my user, pdurbin.
Oliver Bertuch (Sep 19 2023 at 18:44):Interesting - when I mount the file to /tmp, it doesn't work on my machine either!
Oliver Bertuch (Sep 19 2023 at 18:44):It did work when I mounted to /
Oliver Bertuch (Sep 19 2023 at 18:45):dev_bootstrap> /scripts/bootstrap.sh: line 72: /tmp/envfile.txt: Permission denied
Philip Durbin 🚀 (Sep 19 2023 at 18:48):When you say / do you really mean / of my whole file system? Or do you mean I should put my envfile.txt in the root of my git clone of dataverse?
Oliver Bertuch (Sep 19 2023 at 18:50):No, I mean / of the container
Philip Durbin 🚀 (Sep 19 2023 at 18:50):please show me a diff
Oliver Bertuch (Sep 19 2023 at 18:54):I suppose the /tmp folder has the sticky bit set.
Oliver Bertuch (Sep 19 2023 at 18:56):Haha yes that is a security feature
Oliver Bertuch (Sep 19 2023 at 18:57):https://stackoverflow.com/a/70894162
Oliver Bertuch (Sep 19 2023 at 18:58):So better not mount to /tmp :see_no_evil:
Philip Durbin 🚀 (Sep 19 2023 at 18:59):I'm happy to try whatever, a diff would help
Oliver Bertuch (Sep 19 2023 at 18:59):It certainly explains why the checks did not fail but the permission still was denied
Oliver Bertuch (Sep 19 2023 at 18:59):Just remove /tmp
Oliver Bertuch (Sep 19 2023 at 18:59):All of the above but s#/tmp##
Philip Durbin 🚀 (Sep 19 2023 at 19:00):like this?
diff --git a/docker-compose-dev.yml b/docker-compose-dev.yml
index ab44dbc180..1d155561ca 100644
--- a/docker-compose-dev.yml
+++ b/docker-compose-dev.yml
@@ -40,7 +40,11 @@ services:
restart: "no"
command:
- bootstrap.sh
+ - -e
+ - envfile.txt
- dev
+ volumes:
+ - ./envfile.txt:envfile.txt
networks:
- dataverse
trying it
Philip Durbin 🚀 (Sep 19 2023 at 19:04):[ERROR] DOCKER> I/O Error [Unable to create container for [gdcc/configbaker:unstable] : {"message":"invalid volume specification: '/Users/pdurbin/github/iqss/dataverse/envfile.txt:envfile.txt': invalid mount config for type \"bind\": invalid mount path: 'envfile.txt' mount path must be absolute"} (Internal Server Error: 500)]
Philip Durbin 🚀 (Sep 19 2023 at 19:05):haven't given up yet
Oliver Bertuch (Sep 19 2023 at 19:09):Dude you deleted both / now
Oliver Bertuch (Sep 19 2023 at 19:10):Plz go for /envfile.txt
Philip Durbin 🚀 (Sep 19 2023 at 19:10):got it!
diff --git a/docker-compose-dev.yml b/docker-compose-dev.yml
index ab44dbc180..4a0d40917e 100644
--- a/docker-compose-dev.yml
+++ b/docker-compose-dev.yml
@@ -40,7 +40,11 @@ services:
restart: "no"
command:
- bootstrap.sh
+ - -e
+ - envfile.txt
- dev
+ volumes:
+ - ./envfile.txt:/envfile.txt
networks:
- dataverse
$ cat envfile.txt
API_TOKEN=97cad752-983a-446f-866e-b032998ee33d
Splendid!
Philip Durbin 🚀 (Sep 19 2023 at 19:12):That's what you wanted me to do?
Oliver Bertuch (Sep 19 2023 at 19:12):Yes Sir! :grinning:
Philip Durbin 🚀 (Sep 19 2023 at 19:12):phew
Oliver Bertuch (Sep 19 2023 at 19:14):So what do we do now...?
Oliver Bertuch (Sep 19 2023 at 19:14):We're stuck a little with trivial changes to configbaker
Philip Durbin 🚀 (Sep 19 2023 at 19:14):I just approved #9935.
Philip Durbin 🚀 (Sep 19 2023 at 19:15):Now we just wait for that PR to get merged, right?
Oliver Bertuch (Sep 19 2023 at 19:15):Yeah. Or PRs.
Philip Durbin 🚀 (Sep 19 2023 at 19:16):keep 'em coming!
Oliver Bertuch (Sep 19 2023 at 19:16):I could hack on the action some more but I'm stuck because I don't have the upstream changes merged
Philip Durbin 🚀 (Sep 19 2023 at 19:19):Don't worry, there are only 14 PRs in QA or ready for QA! :sweat_smile:
Oliver Bertuch (Sep 19 2023 at 19:20):I'm not sure if our containers-only PRs actually are helpful consuming space in QA
Philip Durbin 🚀 (Sep 19 2023 at 19:22):Sounds like its own topic. :big_smile:
Oliver Bertuch (Sep 19 2023 at 19:30):True, true
Oliver Bertuch (Sep 19 2023 at 19:31):This topic is kinda strange already. We're talking containers in the Python stream... :see_no_evil:
Philip Durbin 🚀 (Sep 19 2023 at 19:31):/me looks
d'oh!
Oliver Bertuch (Sep 19 2023 at 19:32):It's a cross section topic
Philip Durbin 🚀 (Sep 19 2023 at 19:33):
Oliver Bertuch (Sep 20 2023 at 06:48):As this is a cross section topic, let's move this to #dev instead of the Python channel...
Notification Bot (Sep 20 2023 at 06:48):This topic was moved here from #python > Dataverse GitHub Action by Oliver Bertuch.
Oliver Bertuch (Sep 20 2023 at 06:49):Should we brainstorm about a list of inputs and outputs?
Oliver Bertuch (Sep 20 2023 at 06:50):We should probably have an input for the tag and the image names
Oliver Bertuch (Sep 20 2023 at 06:50):Should we have inputs for a Dataverse ref (tag/commit) and build images from there?
Jan Range (Sep 20 2023 at 06:51):Port would also be good
Oliver Bertuch (Sep 20 2023 at 06:51):That way people could use it with forks as well or simply make sure a specific DV version is used
Oliver Bertuch (Sep 20 2023 at 06:52):Port for DV only or also for Solr/Postgres?
Oliver Bertuch (Sep 20 2023 at 06:52):Oh and is that input only or outputs, too?
Jan Range (Sep 20 2023 at 06:53):I'd say yes. Guess its a rare occasion but in case the default ports are occupied already, users can change it.
Jan Range (Sep 20 2023 at 06:54):Oliver Bertuch schrieb:
Oh and is that input only or outputs, too?
Do you mean outputting the ports?
Oliver Bertuch (Sep 20 2023 at 06:54):Yes. Or maybe even provide a URL where DV can be reached once you set it up with the action
Oliver Bertuch (Sep 20 2023 at 06:55):I mean that is probably sth people want - interact with Dataverse
Jan Range (Sep 20 2023 at 06:58):Great idea!
Oliver Bertuch (Sep 20 2023 at 06:59):Input for bootstrap timeout
Oliver Bertuch (Sep 20 2023 at 06:59):Input for bootstrap persona
Jan Range (Sep 20 2023 at 07:01):You've mentioned memory usage too. Would that be one to consider adding?
Oliver Bertuch (Sep 20 2023 at 07:10):Absolutely! Some actions might require more than 2GB RAM
Oliver Bertuch (Sep 20 2023 at 20:18):@Jan Range https://github.com/gdcc/dataverse-action/issues/3
Philip Durbin 🚀 (Sep 20 2023 at 20:20):#9935 has been merged! :tada:
Oliver Bertuch (Sep 20 2023 at 21:52):Progress...
Under https://github.com/gdcc/dataverse-action/actions/runs/6254427827/job/16981926546 you can see that we already are able to gather some insights and inputs.
Cleaned up https://github.com/gdcc/dataverse-action/blob/main/action.yml and the compose file already!
Oliver Bertuch (Sep 20 2023 at 21:53):Also making use of the metadata for postgres/solr version, but keep it overrideable from the action config
Philip Durbin 🚀 (Sep 20 2023 at 23:29):go go go! 
Oliver Bertuch (Sep 21 2023 at 08:14):Added option to supply your own JVM options (can also be set/updated after the containers started!)
Oliver Bertuch (Sep 21 2023 at 08:33):And added the exposing mechanism added with https://github.com/IQSS/dataverse/issues/9935...
Oliver Bertuch (Sep 21 2023 at 08:34):Probably time to start creating some outputs...
Philip Durbin 🚀 (Nov 17 2023 at 14:17):If people want to talk about the action, perhaps because they have a question or want to improve it, where should they go? Can we say something about this in the README? I just opened https://github.com/gdcc/dataverse-action/issues/6 about this.
Philip Durbin 🚀 (Dec 06 2023 at 21:00):Last week we spent a few minutes talking about the action and clean up the issues a bit at https://github.com/gdcc/dataverse-action/issues
Philip Durbin 🚀 (Dec 06 2023 at 21:00):@Jan Range is there any issue or other task we can help with?
Philip Durbin 🚀 (Dec 06 2023 at 21:01):Should we talk about it during tomorrow's meeting?
Jan Range (Dec 06 2023 at 21:05):I'm happy to join tomorrow's meeting :blush: One thing that @Oliver Bertuch already addressed here is to specify a version of Dataverse. Perhaps we can utilize this to some extent to enhance the testing process of pyDataverse.
Philip Durbin 🚀 (Dec 06 2023 at 21:14):Yes! Great stuff happening. Ok! I added it to the agenda. Thanks!
Jan Range (Jan 08 2024 at 09:05):I am currently setting up tests for python-dvuploader and I am using a local installation via docker-compose of the current develop branch. This stack containes S3-compatible services minio and `localstack. It would be very convenient to also have these included in the GitHub Action for CI/CD tests.
If we decide to include both minio and localstack, should they be included by default or made optional? If we choose to make them optional, how can we include them in the Action?
Jan Range (Jan 08 2024 at 09:35):A challenge might be to set/change a collection's storage. Does the endpoint $SERVER_URL/api/dataverses/$PARENT allow to specify a storage location, given the client is a superuser?
Oliver Bertuch (Jan 08 2024 at 09:46):Would it make sense to have a flavor kind of thing going?
Oliver Bertuch (Jan 08 2024 at 09:46):We can include a base compose file with the essential services for Dataverse core
Oliver Bertuch (Jan 08 2024 at 09:47):And add more compose files as flavors with extension that would reference the base file
Oliver Bertuch (Jan 08 2024 at 09:47):A user could choose among the flavors, depending on their use case
Jan Range (Jan 08 2024 at 09:50):Sounds great! I think that's also nice for others who want to deploy DV to re-use these compose files :raised_hands:
Oliver Bertuch (Jan 08 2024 at 09:51):https://docs.docker.com/compose/compose-file/14-include/
Oliver Bertuch (Jan 08 2024 at 09:52):Note: I'm not sure since which Compose version this is included...
Oliver Bertuch (Jan 08 2024 at 09:52):The docs for this have been created ~7 months ago, so it is probably only included in very recent versions of Docker
Oliver Bertuch (Jan 08 2024 at 09:53):(See https://github.com/compose-spec/compose-spec/commit/27008f85ac127e4d91e89bcad8bf7686ffde241f)
Jan Range (Jan 08 2024 at 09:54):Fancy!
Oliver Bertuch (Jan 08 2024 at 09:54):But at least for the Action, we shall be safe if the Docker version included in the runners is recent enough
Oliver Bertuch (Jan 08 2024 at 09:55):It is at least a controlled environment
Jan Range (Jan 08 2024 at 09:55):Off topic: Is there a reason not to use 3 We are using 2.4 atm
Oliver Bertuch (Jan 08 2024 at 09:55):Yes. Docker Maven Plugin compatibility
Jan Range (Jan 08 2024 at 09:55):Ah alright, that makes sense
Oliver Bertuch (Jan 08 2024 at 09:56):We probably should leave a note in the file about that...
Oliver Bertuch (Jan 08 2024 at 09:56):(In upstream - downstream we can do whatever we want)
Jan Range (Jan 08 2024 at 09:57):When this is sorted, how can we create a collection that uses the other storage?
Oliver Bertuch (Jan 08 2024 at 09:57):The easiest way for that would probably be a different bootstrapping flavor
Oliver Bertuch (Jan 08 2024 at 09:58):That will simply require us to provide such a script as a mounted file into the bootstrapping container
Jan Range (Jan 08 2024 at 09:58):Ok, thats nice!
Oliver Bertuch (Jan 08 2024 at 09:59):Yeah the idea behind configbaker was always to keep things modular
Jan Range (Jan 08 2024 at 10:04):
Jan Range (Jan 08 2024 at 10:05):I will create a PR this week to test the flavoring :raised_hands:
Philip Durbin 🚀 (Jan 08 2024 at 15:16):Delicious! :yum:
Philip Durbin 🚀 (Jan 26 2024 at 15:50):I'm including the GitHub Action in a list of ways to run Dataverse in containers (#10273). What use cases am I forgetting?
Use cases for the GitHub Action include:
Testing Client Libraries that interact with Dataverse APIs
Testing Integrations of third party software with Dataverse
Oliver Bertuch (Jan 26 2024 at 19:21):Would metadata blocks count as third party software?
Oliver Bertuch (Jan 26 2024 at 19:21):What about testing Dataverse itself for compliance with some standard?
Oliver Bertuch (Jan 26 2024 at 19:22):Is an SPI extension just an integration with third party software?
Philip Durbin 🚀 (Jan 26 2024 at 19:24):Well, I already have a page for metadata blocks: https://dataverse-guide--10273.org.readthedocs.build/en/10273/container/running/metadata-blocks.html ... but let's discuss that over at #containers > tutorial for demo or eval #10238
Philip Durbin 🚀 (Jan 26 2024 at 19:24):Hmm, like if Dataverse complies with OAI-PMH?
Philip Durbin 🚀 (Jan 26 2024 at 19:31):For SPI stuff, yeah I could maybe see "Testing custom exports" as a potential use of the action.
https://github.com/gdcc/dataverse-exporters contains a "MyJSON" exporter as an example. Maybe we could write a CI workflow that builds the jar file, drops it into the right place, spins up Dataverse, and makes an assertion (look for "myjson" or whatever).
Philip Durbin 🚀 (Jan 26 2024 at 19:32):Anyway, we could discuss any of this over in the other thread. I just wanted to let folks working on the action know that I'm writing about it.
Last updated: Nov 01 2025 at 14:11 UTC