Oliver Bertuch (Mar 16 2023 at 12:43): Oliver Bertuch (Mar 16 2023 at 12:43):Question: do we really want to reuse the scripting via setup-all.sh or do we want sth. else, that might be more helpful?
Oliver Bertuch (Mar 16 2023 at 12:44):I'm thinking along the lines of having a configuration file and some script that makes the necessary calls.
Philip Durbin ๐ (Mar 16 2023 at 12:45):I'm open to change.
Philip Durbin ๐ (Mar 16 2023 at 13:21):Would we have to maintain two scripts doing similar things?
Oliver Bertuch (Mar 16 2023 at 17:06):Yeah, probably. It might be worth considering replacing the setup-all scripts with this, making it reusable for both
Oliver Bertuch (Mar 16 2023 at 17:07):What would happen if we even would be so blunt to make this part of the app?
Oliver Bertuch (Mar 16 2023 at 17:08):E.g. roles... if you really need some custom ones, use the API. But probably most stick with the default ones.
Oliver Bertuch (Mar 16 2023 at 17:09):Same for root dataverse and first admin. Do we really need to use the API for this?
Oliver Bertuch (Mar 16 2023 at 17:11):Same for database settings. We could read them from some file and load them in a Flyway migration
Oliver Bertuch (Mar 16 2023 at 17:12):And so on and so forth
Philip Durbin ๐ (Mar 16 2023 at 17:55):Can we small chunk this? Start with roles? That's the one you mentioned first.
Philip Durbin ๐ (Mar 16 2023 at 19:03):Q: How do you eat an elephant? :elephant:
A: One bite at a time. :eating_utensils:
Philip Durbin ๐ (Mar 21 2023 at 20:28):We just had a nice chat about this at tech hours.
We'll see where #9443 takes us. At the very least we'll remove hard coded ports!
No issue for this yet but if we add a new way for developers to run the API tests locally (in the new containers), as discussed, the "requires a sequence to be added to your database" test is expected to fail: https://guides.dataverse.org/en/5.13/developers/testing.html#identifier-generation
Philip Durbin ๐ (Mar 28 2023 at 19:00):For the test requiring a sequence in postgres: https://dataverse.zulipchat.com/#narrow/stream/375812-containers/topic/running.20API.20tests/near/345214694
Oliver Bertuch (May 04 2023 at 07:11):Alrighta lets make some serious business here. To make integration testing easier, people (@Jan Range) are asking for a Github Action and it's not a good idea to always clone the whole repo just for that. We wanted some config tool / image anyway, so let's hack together some stuff out of what we have, ship that for immediate reuse and go on from there with improvement.
Philip Durbin ๐ (May 04 2023 at 11:18):Sure, but what are we shipping? A larger version of dvinstall.zip? Or something new?
Oliver Bertuch (May 04 2023 at 13:14):In my head I have an image that contains just the bare necessities. So the necessary scripts, the data for those and the solr config parts
Oliver Bertuch (May 04 2023 at 13:14):Make it a teeny tiny alpine based image to execute all the tasks
Oliver Bertuch (May 04 2023 at 13:14):Have some nicely named shellscripts to run as entrypoints
Oliver Bertuch (May 04 2023 at 13:15):Done
Oliver Bertuch (May 04 2023 at 13:15):Anyone can extend that with custom logic
Oliver Bertuch (May 04 2023 at 13:15):We can improve the scripting part
Oliver Bertuch (May 04 2023 at 13:15):And add nice stuff like mdbtool once ready
Oliver Bertuch (May 04 2023 at 13:16):This is kind of a small helper utility thing, bundling all the shenanigans
Oliver Bertuch (May 04 2023 at 13:16):So it get's easy to run those on any platform and skip cloning a repo
Oliver Bertuch (May 04 2023 at 13:17):Sounds like a plan?
Oliver Bertuch (May 04 2023 at 13:18):This is kind of a mixin milestone B and C into one solution
Oliver Bertuch (May 04 2023 at 13:19):Btw what is a good name for this image? gdcc/config? gdcc/init? gdcc/boot?
Philip Durbin ๐ (May 04 2023 at 13:24):Ok, so a new image. In a previous meeting I think we called this a configuration image. Or a bootstrapping image. I kind of like config better.
Jan Range (May 04 2023 at 16:31):@Oliver Bertuch for testing purposes, I think this is a great solution!
Philip Durbin ๐ (May 04 2023 at 16:34):For now we created a note/draft at https://github.com/orgs/IQSS/projects/34/views/17 called "configurator image" that we'll convert into an issue.
Oliver Bertuch (May 04 2023 at 16:55):Let's call the image confighubbub :zany_face:
Philip Durbin ๐ (May 04 2023 at 20:00):I converted the note/draft to an issue: configurator image for container setup #9573
Philip Durbin ๐ (May 04 2023 at 20:00):"setup image" is shorter
Philip Durbin ๐ (May 04 2023 at 20:00):but I'm fine with "configurator image"
Oliver Bertuch (May 04 2023 at 20:01):I'm very interested in the naming of the image itself
Oliver Bertuch (May 04 2023 at 20:01):What shall it be
Oliver Bertuch (May 04 2023 at 20:02): Oliver Bertuch (May 04 2023 at 20:04):Getting more silly be the minute. Please stop me and gimme sth to work with thank you
Oliver Bertuch (May 04 2023 at 20:04):@Jan Range you're online! Please vote!!!
Oliver Bertuch (May 04 2023 at 20:05):Where's @Philip Durbin when I need one? :innocent:
Oliver Bertuch (May 04 2023 at 20:05):@Sherry Lake what does not sound odd to you when you read it? Have your say! :smile:
Jan Range (May 04 2023 at 20:05):Oooh I am the worst at naming things :sweat_smile: Configbaker sounds fancy, but Bootstrapper is nice too
Oliver Bertuch (May 04 2023 at 20:07):You got off (too?) easy when you named EasyDataverse haha @Jan Range
Jan Range (May 04 2023 at 20:08):Yea that was an easy one (no pun intended :-D)
Oliver Bertuch (May 04 2023 at 20:09):There was a pun in there? Didn't notice. :innocent:
Oliver Bertuch (May 04 2023 at 20:09)::beers:
Philip Durbin ๐ (May 04 2023 at 20:09):lol configbaker
Philip Durbin ๐ (May 04 2023 at 20:10)::birthday:
Oliver Bertuch (May 04 2023 at 20:10):Haha looks like we got ourselves a name then
Jan Range (May 04 2023 at 20:10):Edit: Nevermind :grinning:
Oliver Bertuch (May 04 2023 at 20:11):Well certainly Season 3 went :face_palm: at halftime...
Oliver Bertuch (May 04 2023 at 20:12):We better don't follow that path. There be dragons
Philip Durbin ๐ (May 04 2023 at 20:12):When Harvard Dataverse ran on bare metal the servers were called kirk and picard.
Oliver Bertuch (May 04 2023 at 20:13):Isn't there a theory about servers being named after LoR, ST, SW like 80% of the time?
Jan Range (May 04 2023 at 20:13)::spock:
Philip Durbin ๐ (May 04 2023 at 20:13):the old Odyssey cluster at Harvard was full of Greek names for machines
Jan Range (May 04 2023 at 20:13):Ours is called Flanders, because our group leader literally looks like him :sweat_smile:
Oliver Bertuch (May 04 2023 at 20:14):How about gdcc/mando? It's gonna weild a fancy blade, have a neat youngling and shout "This is the way" anytime you use it
Philip Durbin ๐ (May 04 2023 at 20:14):maybe start writing the README and a name will emerge
Philip Durbin ๐ (May 04 2023 at 20:14):what's it going to look like on Docker Hub? the README, I mean
Philip Durbin ๐ (May 04 2023 at 20:15):and will there be a :birthday: emoji?
Oliver Bertuch (May 04 2023 at 20:15):OK lets call it working title "configbaker" for now :innocent:
Oliver Bertuch (May 04 2023 at 20:15):Probably it will be very similar to the other READMEs
Oliver Bertuch (May 04 2023 at 20:15):We can decide to add more "how to use" in it as well
Oliver Bertuch (May 04 2023 at 20:15):Like name available scripts you can use
Philip Durbin ๐ (May 04 2023 at 20:16):You can call it mando as long as the README is clear :happy:
Oliver Bertuch (May 04 2023 at 20:16):This might be helpful not needing to dig into the guide every time for that
Philip Durbin ๐ (May 04 2023 at 20:16):configbaker sounds like a good codename to start with :birthday:
Oliver Bertuch (May 04 2023 at 20:16):Oh BTW it will have a help message printed when you "just run it"
Oliver Bertuch (May 04 2023 at 20:16):That probably should list all the scripts baked in
Philip Durbin ๐ (May 04 2023 at 20:17):I added a :birthday: to the description of #9573. That's my contribution so far. :happy:
Oliver Bertuch (May 04 2023 at 20:23):
Philip Durbin ๐ (May 04 2023 at 20:33):How about "it can cook up an instance"
Oliver Bertuch (May 04 2023 at 20:33):Baking now involves cooking?
Philip Durbin ๐ (May 04 2023 at 20:34):we'll workshop it some more :sweat_smile:
Oliver Bertuch (May 04 2023 at 20:34):
Philip Durbin ๐ (May 04 2023 at 20:35):oh, sorry, I meant instead of bootstrap, the same sentence with icing
Oliver Bertuch (May 04 2023 at 20:36):echo "Hello!"
echo ""
echo "This is ConfigBaker, a container image with lots of tooling to cook up a containerized Dataverse instance."
echo "It can bootstrap an instance (initial config), put icing on your Solr Search Index configuration and more."
echo ""
echo "Here's a list of command scripts available:"
Please rephrase to your liking :wink:
Philip Durbin ๐ (May 04 2023 at 20:42):echo "Hello!"
echo ""
echo "I'm ConfigBaker, a container image with lots of tooling to "bake" a containerized Dataverse instance!"
echo "I can cook up an instance (initial config), put icing on your Solr search index configuration, and more!"
echo ""
echo "Here's a list of things I can do for you:"
everybody wants to talk to bots these days so I say we go with it
Oliver Bertuch (May 04 2023 at 20:52):Haha make ChatGPT setup a Dataverse instance for ya
Oliver Bertuch (May 04 2023 at 20:53):What happens when you ask it that?
Philip Durbin ๐ (May 04 2023 at 21:03):It tells you to RTFM:
Screen-Shot-2023-05-04-at-5.02.21-PM.png
Oliver Bertuch (May 04 2023 at 21:03):Hahahahahahahahahaha
Oliver Bertuch (May 04 2023 at 21:03):Utter nonsense
Philip Durbin ๐ (May 04 2023 at 21:06):It ends with this:
"If you are not familiar with server administration or installing software, it is recommended that you seek the assistance of a qualified professional to help you with the installation process."
@Don Sizemore help!
Oliver Bertuch (May 04 2023 at 21:34):Alright got a script that does the file permission fixing for us
Oliver Bertuch (May 04 2023 at 21:34):I should create some commits and push this to some place as an image
Philip Durbin ๐ (May 04 2023 at 21:36):yes!
Oliver Bertuch (May 04 2023 at 21:39):Feel adventurous? Go try https://hub.docker.com/r/gdcc/configbaker
Oliver Bertuch (May 04 2023 at 21:40):Here's a command for you: docker run -it --rm gdcc/configbaker:unstable
Oliver Bertuch (May 04 2023 at 21:49):So there now is also https://github.com/IQSS/dataverse/pull/9574. Quite naked so far, but at least we started working on it!
Oliver Bertuch (May 04 2023 at 21:49):I'm gonna close the hacking session now for today. It's been exciting! See y'all tomorrow
Oliver Bertuch (May 04 2023 at 22:10):OK I lied - I added printing a script description to the helper script because this is user friendly (but extracting it from the script files, so this is not hardcoded...)
Oliver Bertuch (May 04 2023 at 22:11):
Philip Durbin ๐ (May 04 2023 at 22:51):Works great. I love how small it is!
pdurbin@air ~ % docker run -it --rm gdcc/configbaker:unstable
Unable to find image 'gdcc/configbaker:unstable' locally
unstable: Pulling from gdcc/configbaker
c41833b44d91: Pull complete
6afcb51be105: Pull complete
8d4c0b2520e6: Pull complete
3727a8da81b4: Pull complete
Digest: sha256:47498c6d054bb4b24f13a1b3659ffe18aedc08183c1afdb1462c9f63d2d12496
Status: Downloaded newer image for gdcc/configbaker:unstable
โmฮฑo
โซ jh
`%โฅรฆโจ
โซยต
โ@M%โ,
โ` โซU
โยฒ โซโ
โM#Mโ"
โรฆMโโ%ฯโซโ
โโซ" "โซโ
โ โ
โ โ
`โซยต ยฟโซ"
"โ%%MMโ`
Hello!
I'm ConfigBaker, a container image with lots of tooling to 'bake' a containerized Dataverse instance!
I can cook up an instance (initial config), put icing on your Solr search index configuration, and more!
Here's a list of things I can do for you:
fix-fs-permissions.sh - Fix folder permissions using 'chown' to be writeable by containers not running as root.
help.sh - This script.
Simply execute this container with the script name (and potentially arguments) as 'command'.
pdurbin@air ~ %
Ok here's the first tricky question. Do we provide a full configset for Solr already or don't we?
Oliver Bertuch (May 08 2023 at 15:01):Providing it means very simply start command for Solr, needs copy hacking otherwise. (As we do now)
Philip Durbin ๐ (May 08 2023 at 15:55):I'm fine with copy hacking for now. We can remove it later.
Oliver Bertuch (May 09 2023 at 13:15):I'm having trouble with the compose file, the maven docker plugin and the new bakery... :sad:
Oliver Bertuch (May 09 2023 at 13:15):This stuff doesn't really play along well
Oliver Bertuch (May 09 2023 at 13:16):I thought I could make a compose build only a reality for this, but it beats up the Maven side completely
Oliver Bertuch (May 09 2023 at 13:16)::pensive:
Oliver Bertuch (May 09 2023 at 13:24):This is kinda frustrating. I probably will go for a Maven build then first.
Philip Durbin ๐ (May 09 2023 at 13:25):Sure. Sounds good.
Oliver Bertuch (May 09 2023 at 13:47):@Philip Durbin is there a way to detect from the outside that an instance is bootstrapped?
Oliver Bertuch (May 09 2023 at 13:47):I'd say the best way to have that is by checking for a root dataverse
Oliver Bertuch (May 09 2023 at 13:47):But that might not be published yet
Oliver Bertuch (May 09 2023 at 13:48):Would it make sense to add a /api/info/initialized thingy?
Philip Durbin ๐ (May 09 2023 at 13:49):I'm not opposed to adding an API endpoint for this.
Philip Durbin ๐ (May 09 2023 at 13:50):Checking for the root dataverse could work. @Jan Range was checking for metadata blocks, which get added later.
Philip Durbin ๐ (May 09 2023 at 13:50):Maybe we could check for the very last thing that gets added, whatever that is.
Oliver Bertuch (May 09 2023 at 13:50):Or maybe both
Oliver Bertuch (May 09 2023 at 13:50):It definitely needs both things
Oliver Bertuch (May 09 2023 at 13:51):OK maybe for now I'll stick to the blocks thing, thx for the idea @Jan Range
Philip Durbin ๐ (May 09 2023 at 13:52):These are the last two things the final setup script configures:
echo "Publishing root dataverse..."
curl -H "X-Dataverse-key:$API_TOKEN" -X POST "http://localhost:8080/api/dataverses/:root/actions/:publish"
echo "Allowing users to create dataverses and datasets in root..."
curl -H "X-Dataverse-key:$API_TOKEN" -X POST -H "Content-type:application/json" -d "{\"assignee\": \":authenticated-users\",\"role\": \"fullContributor\"}" "http://localhost:8080/api/dataverses/:root/assignments"
Oliver Bertuch (May 09 2023 at 13:52):Yeah but I want sth more general ;-)
Philip Durbin ๐ (May 09 2023 at 13:52):sure
Oliver Bertuch (May 09 2023 at 13:53):I'm already adding personas so we have it easier to add more bootstrapping things
Oliver Bertuch (May 09 2023 at 13:53):Don't have a commit yet, sry
Oliver Bertuch (May 09 2023 at 13:53):I will touch the setup scripts in scripts/api in this PR
Oliver Bertuch (May 09 2023 at 13:54):No way around this - I simply can't use localhost:8080 from inside a container
Oliver Bertuch (May 09 2023 at 13:54):(The change is minimal - simply replace the static URL with a variable)
Philip Durbin ๐ (May 09 2023 at 13:54):Is it worth including a fix for the issue @Akio Sone opened recently? Configurable server-port-number for the containerized Dataverse #9534
Oliver Bertuch (May 09 2023 at 13:55):Dunno if this means we get in trouble with Kevin and QA
Oliver Bertuch (May 09 2023 at 13:56):I fear @Akio Sone is mixing up a few things here. From inside the container network, we can simply reach out to http://dataverse:8080. There is no need to rewire these ports
Oliver Bertuch (May 09 2023 at 13:56):At least not for configuration
Oliver Bertuch (May 09 2023 at 13:57):If he wants to expose the container ports to some other port on his machine, that must be configured in the compose file.
Philip Durbin ๐ (May 09 2023 at 13:58):Well, his use case is real (maybe a new topic). As a Java developer he has other apps he's developing running on port 8080 (on his laptop or desktop) and would like to run Dataverse on some other port.
Oliver Bertuch (May 09 2023 at 13:58):And by the way - rewiring these internal ports is not as easy as it sounds - this needs to rewired all the way down to the Payara server conf. I don't think that is either necessary nor useful
Oliver Bertuch (May 09 2023 at 13:58):Sure - he may simply change his local compose file
Philip Durbin ๐ (May 09 2023 at 13:58):Just saying it's a real use case. :happy:
Philip Durbin ๐ (May 09 2023 at 14:00):As for QA etc, can configbaker help with the SPA? That's the way to get buy in.
Oliver Bertuch (May 09 2023 at 14:00):https://docs.docker.com/compose/compose-file/compose-file-v2/#ports
Either specify both ports (HOST:CONTAINER), or just the container port (an ephemeral host port is chosen).
Oliver Bertuch (May 09 2023 at 14:00):Yes of course it can!
Oliver Bertuch (May 09 2023 at 14:00):It will simply make calling the bootstrap script unnecessary!
Philip Durbin ๐ (May 09 2023 at 14:00):Ok, let's make sure we talk this in the top of the PR.
Philip Durbin ๐ (May 09 2023 at 14:00):"why we need this"
Oliver Bertuch (May 09 2023 at 14:01):Simply because it can sit there as an init container, started alongside the others when you run compose
Oliver Bertuch (May 09 2023 at 14:01):Good point!
Oliver Bertuch (May 09 2023 at 14:02):OK I'll go ahead and make some commits so you folks can see some results
Philip Durbin ๐ (May 09 2023 at 14:02):We could even wordsmith #9573 around the SPA. Maybe even put it in the SPA column if we do a good job selling it. :birthday:
Philip Durbin ๐ (May 09 2023 at 14:05):possible new titles:
Philip Durbin ๐ (May 09 2023 at 14:06): Philip Durbin ๐ (May 09 2023 at 14:14):Do they get any new features?
Philip Durbin ๐ (May 09 2023 at 14:14):Or is the main thing that they don't have to run the final setup script?
Oliver Bertuch (May 09 2023 at 14:15):Yeah that is the main thing
Philip Durbin ๐ (May 09 2023 at 14:15):Ok. Small chunks are good!
Oliver Bertuch (May 09 2023 at 14:16):OK I just pushed the changes to the setup scripts
Oliver Bertuch (May 09 2023 at 14:16):It removes the necessity to cd to the script's dir first and adds the override cap
Philip Durbin ๐ (May 09 2023 at 14:17):Cool. I'm looking at it (#9574). Did you see my comments from before?
Oliver Bertuch (May 09 2023 at 14:18):Yes but I wanted cosmetics later :yum:
Philip Durbin ๐ (May 09 2023 at 14:25):Ok, I added a review.
Philip Durbin ๐ (May 09 2023 at 14:25):Not much to say. Looks good. Should work.
Philip Durbin ๐ (May 09 2023 at 14:26):I'd probably want to try it with my dev rebuild script.
Philip Durbin ๐ (May 09 2023 at 14:26):That is, outside the context of containers.
Oliver Bertuch (May 09 2023 at 14:27):Sounds good!
Oliver Bertuch (May 09 2023 at 14:28):One can also use the "old script" for initializing containers
Oliver Bertuch (May 09 2023 at 14:28):It makes use of the scripts
Oliver Bertuch (May 09 2023 at 14:28):So this should be easy to testdrive
Philip Durbin ๐ (May 09 2023 at 14:29):Good point.
Philip Durbin ๐ (May 09 2023 at 14:30):And Jenkins will exercise setup-all, I believe.
Oliver Bertuch (May 09 2023 at 14:30):Thank god for @Don Sizemore having us covered!
Philip Durbin ๐ (May 09 2023 at 14:35):
@Don Sizemore
Oliver Bertuch (May 09 2023 at 15:45):Getting closer...
Oliver Bertuch (May 09 2023 at 16:04):Anyone feeling adventurous? I just pushed a new version of configbaker to Docker Hub
Oliver Bertuch (May 09 2023 at 16:05):And it's looking pret-ty good around here with making it bootstrapping things for ya!
Oliver Bertuch (May 09 2023 at 16:13):@Philip Durbin I just pushed all the changes I have so far
Oliver Bertuch (May 09 2023 at 16:13):One should be able to run mvn -Pct clean package docker:run and see the magic happen!
Oliver Bertuch (May 09 2023 at 16:14):Works on my machine... :yum:
Oliver Bertuch (May 09 2023 at 16:15):@Guillermo Portas dunno if you're feeling adventurous as well :wink:
Philip Durbin ๐ (May 09 2023 at 16:21):orly
Oliver Bertuch (May 09 2023 at 16:57):Yeah rly :big_smile:
Philip Durbin ๐ (May 10 2023 at 00:33):I love how friendly it is!
dev_bootstrap> Done, your instance has been configured for development. Have a nice day!
And that it works. Another triumph by @Oliver Bertuch!
Oliver Bertuch (May 10 2023 at 04:56):It's a baker after all :birthday::cake::cookie::donut::pie:
Jan Range (May 10 2023 at 09:12):This should be presented every time
Oliver Bertuch (May 10 2023 at 09:25):I just tried to convert that into Ascii Art... Too many details :wink:
Oliver Bertuch (May 10 2023 at 09:26):But actually I thought about making an ascii art presentation of a cake and have the text nicely added to the side
Jan Range (May 10 2023 at 09:30):This one is maybe more suited for Ascii if we are able to edit it :-D
Oliver Bertuch (May 10 2023 at 11:44):Alright, finished the large review for Jim, onto new adventures! Hit me with things about this baker. After proofing the concept now's the time to extend and polish
Philip Durbin ๐ (May 10 2023 at 11:45):Should we go ahead and merge https://github.com/IQSS/dataverse-frontend/pull/87 and worry about switching to configbaker later?
Oliver Bertuch (May 10 2023 at 11:46):Sure, why not. There will always be room for improvement
Oliver Bertuch (May 10 2023 at 11:46):Except @Guillermo Portas wants to avoid touching it again :wink:
Philip Durbin ๐ (May 10 2023 at 11:47):Ok. I won't pull it out of "ready for QA" then.
Oliver Bertuch (May 10 2023 at 11:47):@Philip Durbin you are today in the luxury position of telling me what to look into next: go for more baking or start a new PoC for a Github Action
Philip Durbin ๐ (May 10 2023 at 11:50):Can you please start a branch for docs? Even a tiny diff helps. https://dataverse.zulipchat.com/#narrow/stream/375812-containers/topic/docs.20for.20milestone.20A.20.239540
Philip Durbin ๐ (May 10 2023 at 13:05):In the backlog, what size should I give #9574?
Oliver Bertuch (May 10 2023 at 13:09):Are you counting your experimentation as well? Then maybe a 10? Otherwise a 3
Philip Durbin ๐ (May 10 2023 at 13:12):A 3 it is, thanks.
Philip Durbin ๐ (May 10 2023 at 13:12):I have docs above it. I hope you don't mind.
Oliver Bertuch (May 10 2023 at 13:18):Meaning you want to see the docs PR done first? No problemo senior!
Oliver Bertuch (May 10 2023 at 13:19):
Philip Durbin ๐ (May 10 2023 at 13:23):Yeah. I'm talking about the column in the backlog. We should put at the top stuff that we want to be put in a sprint.
Philip Durbin ๐ (May 10 2023 at 13:23):Of course, docs do have a fast track option. :thinking:
Philip Durbin ๐ (May 10 2023 at 13:24):I put config baker first. :birthday:
Philip Durbin ๐ (May 10 2023 at 13:24):When will it no longer be draft?
Oliver Bertuch (May 10 2023 at 13:26):Probably we should think about a definition of done for it...
Philip Durbin ๐ (May 10 2023 at 13:27):It seems like it already delivers value. Ship it? :rocket:
Oliver Bertuch (May 10 2023 at 13:27):But it sounds like we should finish the docs for milestone A first so we can call them "done"
Oliver Bertuch (May 10 2023 at 13:27):Oh no, please let me polish things a bit more
Oliver Bertuch (May 10 2023 at 13:27):And it definitely needs docs
Guillermo Portas (May 10 2023 at 14:24):Yes, we can improve the local frontend environment with the configbaker later, so we can start using it as it is for now :)
Oliver Bertuch (May 10 2023 at 17:59):So what _is_ a definition of done for this?
I think we still need:
That should be it for now, as long as we don't have stuff like mdbtools ready. We could add a helper for auth providers
Philip Durbin ๐ (May 10 2023 at 18:03):Quite a list! :sweat_smile:
Philip Durbin ๐ (May 10 2023 at 18:49):I added :birthday: configbaker :birthday: to tomorrow's agenda: https://docs.google.com/document/d/1eQVm88dP2rgM9DKn4ivoWBx6MOK6aXfkLhsZN-Y3fsc/edit?usp=sharing
Oliver Bertuch (May 10 2023 at 18:53):"Brought to you by your local German baker" (Pun intended)
Philip Durbin ๐ (May 25 2023 at 14:58):As we just discussed, #9574 is still draft. Let's make it non-draft so I can advocate to get it tested and merged. How can I help?
Oliver Bertuch (May 25 2023 at 15:06):Hmm maybe draft some docs?
Philip Durbin ๐ (May 25 2023 at 15:13):Should I push them directly to your branch?
Philip Durbin ๐ (May 25 2023 at 15:13):Also, I just brought this up at standup. I think we can get it in.
Oliver Bertuch (May 25 2023 at 15:28):Great!
Oliver Bertuch (May 25 2023 at 15:29):Yes, please feel free to push into my branch
Oliver Bertuch (May 25 2023 at 15:29):That's probably the easiest way forward
Philip Durbin ๐ (May 25 2023 at 17:45):@Oliver Bertuch do you want to keep the support email as-is? https://github.com/IQSS/dataverse/pull/9574/files#r1185609395
Philip Durbin ๐ (May 25 2023 at 17:46):$ ack forschungsdaten
modules/container-configbaker/Dockerfile
47: org.opencontainers.image.authors="Research Data Management at FZJ <forschungsdaten@fz-juelich.de>" \
modules/container-base/src/main/docker/Dockerfile
225: org.opencontainers.image.authors="Research Data Management at FZJ <forschungsdaten@fz-juelich.de>" \
src/main/docker/Dockerfile
45: org.opencontainers.image.authors="Research Data Management at FZJ <forschungsdaten@fz-juelich.de>" \
I'm fine with whatever. I'm somewhat obligied to mark things I did but sure, we can change those.
Oliver Bertuch (May 25 2023 at 17:51):To what BTW
Oliver Bertuch (May 25 2023 at 17:51):Maybe we should start using the REUSE framework
Philip Durbin ๐ (May 25 2023 at 18:07):ok
Philip Durbin ๐ (May 25 2023 at 18:07):I'm having trouble building the configbaker image.
Philip Durbin ๐ (May 25 2023 at 18:07):I was trying to document it.
Philip Durbin ๐ (May 25 2023 at 18:08):no pom.xml file
Philip Durbin ๐ (May 25 2023 at 18:09):I guess I'll just remove Build Instructions
Philip Durbin ๐ (May 25 2023 at 18:15):@Oliver Bertuch ok, I pushed some docs: #9574
Philip Durbin ๐ (May 25 2023 at 18:16):Is it ready for QA? Or should we also add a GitHub Action?
Philip Durbin ๐ (May 25 2023 at 18:16):... to build the image and push it to registries, I mean
Oliver Bertuch (May 25 2023 at 20:05):Absolutely, we should have a CI workflow for that around.
Philip Durbin ๐ (May 25 2023 at 20:08):Before we send it to QA?
Oliver Bertuch (May 25 2023 at 20:37):Yes please!
Oliver Bertuch (May 25 2023 at 20:37):Sorry, just back to desk
Oliver Bertuch (May 25 2023 at 20:37):You probably already headed home :wink:
Philip Durbin ๐ (May 25 2023 at 20:45):Nope. 15 minutes left. Need to bake a real cake tonight. The older kiddo has a birthday tomorrow.
Philip Durbin ๐ (May 25 2023 at 20:46):I'd say getting the images pushed is the priority but I did just leave a comment about SOLR_URL here: https://github.com/IQSS/dataverse/pull/9574/files#r1205986701
Oliver Bertuch (May 25 2023 at 20:47):I'd say you can simply ignore that one
Philip Durbin ๐ (May 25 2023 at 20:47):And maybe more docs. I dunno. I added some, at least. Probably good enough.
Oliver Bertuch (May 25 2023 at 20:48):To me this line doesn't make much sense. When I'm setting up Dataverse, the index should be empty.
Oliver Bertuch (May 25 2023 at 20:48):Why would I delete it before going on?
Oliver Bertuch (May 25 2023 at 20:48):Maybe this was added to the installer when someone wants to use it to start fresh
Philip Durbin ๐ (May 25 2023 at 20:48):It's just a legacy line from my "drop database and start over" script.
Oliver Bertuch (May 25 2023 at 20:49):But then I'm not sure why it wasn't added to that drop and rebuild script
Oliver Bertuch (May 25 2023 at 20:49):Well we could be bold and just say drop the line
Philip Durbin ๐ (May 25 2023 at 20:50):Meh. I vote later.
Oliver Bertuch (May 25 2023 at 20:51):You're just afraid of QA :melting_face:
Philip Durbin ๐ (May 25 2023 at 20:51):Just trying to keep PRs small. :upside_down:
Oliver Bertuch (May 25 2023 at 20:52):Huh? The line is already being changed and the SOLR_URL defined as an additional line. Deleting the line would actually be of the same size or less?
Oliver Bertuch (May 25 2023 at 20:52):Or are you referring to mental load PR size?
Philip Durbin ๐ (May 25 2023 at 20:53):I'm always trying to avoid questions and discussion. :happy:
Oliver Bertuch (May 25 2023 at 20:53):Ha now I know why this codebase is full of tech debt :stuck_out_tongue_closed_eyes: :wink:
Oliver Bertuch (May 25 2023 at 20:55):OK here's our todo list for configbaker again:
Philip Durbin ๐ (May 25 2023 at 20:55):Wow, I added that "clear out Solr" line back when we were still indexing users: https://github.com/IQSS/dataverse/commit/42692f5681ee65ea982b66656d7683bf0e015725
Oliver Bertuch (May 25 2023 at 20:56):It's been sitting there for a while :smiley:
Philip Durbin ๐ (May 25 2023 at 20:56):I can remove it if you want.
Oliver Bertuch (May 25 2023 at 20:57):Or shall I do that commit? Then it's me to blame :see_no_evil:
Philip Durbin ๐ (May 25 2023 at 20:57):No, I got it. One sec.
Oliver Bertuch (May 25 2023 at 20:57):You'd have believable excuse: "I told him so!"
Philip Durbin ๐ (May 25 2023 at 20:59):ok, pushed
Philip Durbin ๐ (May 25 2023 at 21:00):For the bullet above about docs, I did add some.
Oliver Bertuch (May 25 2023 at 21:00):Yeah I'm just about to read through those! Lovely!
Philip Durbin ๐ (May 25 2023 at 21:01):I couldn't figure out how to build the image.
Oliver Bertuch (May 25 2023 at 21:01):It's built together with the app image
Oliver Bertuch (May 25 2023 at 21:01):Because they are tightly coupled
Oliver Bertuch (May 25 2023 at 21:02):If you change some scripts etc you need to use that with the according app image
Philip Durbin ๐ (May 25 2023 at 21:02):That's sort of what I figured. Is it possible to build them separately? If so, should we document it?
Oliver Bertuch (May 25 2023 at 21:02):Yeah that should be possible - you can filter the images.
Philip Durbin ๐ (May 25 2023 at 21:03):Ok. If you feel like testing and documenting if, please feel free, but I'd say it's a low priority.
Oliver Bertuch (May 25 2023 at 21:03):OK.
Oliver Bertuch (May 25 2023 at 21:04):I think I'll start by merging in develop and then take a look at the compose file, get that done.
Philip Durbin ๐ (May 25 2023 at 21:04):Sounds good. I'd better get home. :birthday:
Oliver Bertuch (May 25 2023 at 21:05):Go bake bake bake away!
Oliver Bertuch (May 25 2023 at 21:05):Good luck!
Oliver Bertuch (May 25 2023 at 21:05):(No fish please)
Oliver Bertuch (May 25 2023 at 22:52):Sorry I didn't get far with Configbaker this evening. I first had to fix https://github.com/IQSS/dataverse/issues/9617. These random test failures are driving me nuts... :angry:
Philip Durbin ๐ (May 26 2023 at 11:07):Thanks! I moved it to QA.
Oliver Bertuch (May 26 2023 at 11:10):Lovely! It hit me again yesterday when hacking on config baker. Wondering y'all are not receiving that much messages from Actions
Philip Durbin ๐ (May 26 2023 at 11:15):Not sure. There have been flakier tests. But good to fix this one. Thanks again.
Oliver Bertuch (May 26 2023 at 11:35):Maybe I just got lucky and had all the failing ones
Philip Durbin ๐ (May 26 2023 at 12:10):Oh, I would have written some docs on how to extend configbaker with your own scripts, but I wasn't sure how this works. Do you simply fork configbaker, add your scripts, commit and push, build your image, maybe push to a registry, and run it?
Philip Durbin ๐ (May 26 2023 at 12:18):Or maybe you put your custom scripts on disk somewhere and point configbaker at them?
Oliver Bertuch (May 26 2023 at 12:19):I wouldn't call it fork. You could use it as a base image. But yes, you could also just mount the scripts
Oliver Bertuch (May 26 2023 at 12:19):This would be especially true for the personas when bootstrapping
Oliver Bertuch (May 26 2023 at 12:20):Either for testing without rebuilding the image (also that is really really fast)
Philip Durbin ๐ (May 26 2023 at 12:20):Ok. Mount the scripts.
Oliver Bertuch (May 26 2023 at 12:20):Or you might want to prepare a persona package in a pipeline (e.g. add special blocks / users / ...) and execute the pipeline's result
Oliver Bertuch (May 26 2023 at 12:21):Both is possible
Oliver Bertuch (May 26 2023 at 12:21):Both makes a lot of sense
Oliver Bertuch (May 26 2023 at 12:21):Depends on your use case
Philip Durbin ๐ (May 26 2023 at 12:23):Ok. It might be nice to document one of these options.
Philip Durbin ๐ (May 26 2023 at 13:54):Any hacking you want me to do on the 9573-configbaker branch? Or should I leave it alone (or hack in a separate branch)?
Oliver Bertuch (May 26 2023 at 14:12):Oh plz go ahead and hack on it. I'm preparing a poster... :-( Deadline coming up fast
Philip Durbin ๐ (May 26 2023 at 14:21):Ok. I'm trying to get some other stuff done too. I'll make noise here if I'm going to start. I promise nothing. :happy:
Oliver Bertuch (May 30 2023 at 19:16): Oliver Bertuch (May 31 2023 at 12:23):@Philip Durbin question: are we fine with configbaker being built alongside the app image? Would it make sense to split that up a little more?
Philip Durbin ๐ (May 31 2023 at 12:38):I'm fine with how it is.
That said, configbaker doesn't have much to do with Java, does it? Perhaps we could just have a simple docker build process instead of using mvn.
Oliver Bertuch (May 31 2023 at 12:48):No we can't.
Oliver Bertuch (May 31 2023 at 12:48):I don't want any more bash glue around docker-compose just to get the files into place etc
Oliver Bertuch (May 31 2023 at 12:49):Native docker is really bad at getting not-aligned-for-Docker collections of files together
Oliver Bertuch (May 31 2023 at 12:50):The workaround is "send the complete project into the context", which means transferring all folder and subfolders during a build
Oliver Bertuch (May 31 2023 at 12:50):(That's slow)
Oliver Bertuch (May 31 2023 at 13:21):@Philip Durbin et al - WDYT https://dataverse-guide--9574.org.readthedocs.build/en/9574/container/configbaker-image.html
Philip Durbin ๐ (May 31 2023 at 13:25):Thanks for adding build instructions!
Philip Durbin ๐ (May 31 2023 at 13:25):And the examples. Mounting scripts for a persona.
Oliver Bertuch (May 31 2023 at 13:28):We can add more as we go and see fit
Oliver Bertuch (May 31 2023 at 13:28):There are a lot of possibilities
Philip Durbin ๐ (May 31 2023 at 13:30):Only docs left, according to the strikeouts above. Want me to hack a bit and then move it to QA?
Oliver Bertuch (May 31 2023 at 13:32):If you want we can have a cowriting session
Oliver Bertuch (May 31 2023 at 13:32):Let's get this done and in the pipeline for Kevin :smile:
Oliver Bertuch (May 31 2023 at 13:33):The leftover things for docs: maybe iterate on the guide page AND update the README on Docker Hub. We need to think about what we want on that page
Philip Durbin ๐ (May 31 2023 at 13:35):I sort of want the README on DockerHub to just say "please read the container guide"
Oliver Bertuch (May 31 2023 at 13:35):Fair enough!
Oliver Bertuch (May 31 2023 at 13:37):I'll start hacking on it a little
Philip Durbin ๐ (May 31 2023 at 13:37):Wait, you're changing if from ConfigBaker to Config Baker?
Oliver Bertuch (May 31 2023 at 13:37):I think it's easier to read that way
Oliver Bertuch (May 31 2023 at 13:37):The image tag is still configbaker because no spaces in those
Philip Durbin ๐ (May 31 2023 at 13:38):Ok, but we should be consistent. I'm hacking on docs locally. Will push.
Philip Durbin ๐ (May 31 2023 at 13:56):@Oliver Bertuch are you ready to make the configbaker PR non-draft?
Oliver Bertuch (May 31 2023 at 13:56):Did you hack on that README?
Oliver Bertuch (May 31 2023 at 13:56):I didn't because you said you're up to sth
Philip Durbin ๐ (May 31 2023 at 14:11):A tiny bit.
Philip Durbin ๐ (May 31 2023 at 14:11):Should we s/dev_bootstrap/dev_configbaker/?
Oliver Bertuch (May 31 2023 at 14:12):I wouldn't. The service is doing the bootstrapping. The solr initializer uses configbaker as well. Let's stick to names of functionality
Oliver Bertuch (May 31 2023 at 14:14):I just pushed a small commit to make things more consistent with Config Baker
Oliver Bertuch (May 31 2023 at 14:14):I think I'm satisfied with this as is for now
Oliver Bertuch (May 31 2023 at 14:14):Flagged as "ready to review"
Philip Durbin ๐ (May 31 2023 at 14:18):Hmm, I'm sort of in git stash hell now. :flame:
Oliver Bertuch (May 31 2023 at 14:19):I see an IT failing... DatasetsIT.testCuratePublishedDatasetVersionCommand
Philip Durbin ๐ (May 31 2023 at 14:20):ah, this helping: https://stackoverflow.com/questions/8515729/how-to-abort-a-stash-pop/60444590#60444590
Oliver Bertuch (May 31 2023 at 14:21):Ha but it wasn't us! Develop is failing with the same result! https://jenkins.dataverse.org/blue/organizations/jenkins/IQSS-dataverse-develop/detail/IQSS-dataverse-develop/1359/pipeline
Philip Durbin ๐ (May 31 2023 at 14:24):Also here: https://github.com/IQSS/dataverse/pull/9558#issuecomment-1523864443
Philip Durbin ๐ (May 31 2023 at 14:24):I left a comment here an hour ago: https://github.com/IQSS/dataverse/pull/9558#issuecomment-1570229407
Philip Durbin ๐ (May 31 2023 at 14:31):Approved! Off to QA.
Philip Durbin ๐ (May 31 2023 at 14:31):I added a quickstart. I hope you don't mind.
Oliver Bertuch (May 31 2023 at 14:40):Ha! How dare you! :yum:
Philip Durbin ๐ (May 31 2023 at 14:48):For people like me that are like "how do I run a container again? I forget"
Philip Durbin ๐ (May 31 2023 at 14:49):Hey, there's already an image at https://hub.docker.com/r/gdcc/configbaker
Philip Durbin ๐ (May 31 2023 at 14:49):22 days old though. I guess we shouldn't tell people to use it.
Oliver Bertuch (May 31 2023 at 14:53):I had to create the repo at Docker Hub to set it up, set permissions etc
Oliver Bertuch (May 31 2023 at 14:54):I can always push a new version manually (like it did with the one already there)
Oliver Bertuch (Jun 01 2023 at 16:45):Configbaker is merged!!!
Oliver Bertuch (Jun 01 2023 at 16:45)::partying_face:
Philip Durbin ๐ (Jun 01 2023 at 20:50)::tada:
Guillermo Portas (Jun 08 2023 at 12:22):Including configbaker in the frontend local dev environment: https://github.com/IQSS/dataverse-frontend/commit/4034defae61d03f7217bf4ee9ff61d6338ccbe0d :wink:
Philip Durbin ๐ (Jun 08 2023 at 14:38):Nice! Less code to maintain! :tada:
Philip Durbin ๐ (Aug 11 2023 at 15:16):We have Config Baker now! https://guides.dataverse.org/en/5.14/container/configbaker-image.html
I'm resolving this topic. Please open new ones for followup.
Notification Bot (Aug 11 2023 at 15:16):Philip Durbin has marked this topic as resolved.
Last updated: Oct 30 2025 at 05:14 UTC