Oliver Bertuch (Mar 15 2023 at 18:37): Oliver Bertuch (Mar 15 2023 at 18:37):With #9439 almost done, one of the next steps is #9444. To play with this, I created https://github.com/poikilotherm/test-image-push-flows, avoiding lots of noise in the main project and faster turn around times.
Oliver Bertuch (Mar 15 2023 at 18:39):This actually works quite well already... See the packages at GHCR and at Docker Hub 1, Docker Hub 2
Oliver Bertuch (Mar 15 2023 at 18:40):One of the crucial parts is to switch the Maven Unittest workflow to not run on push to any branch, but only for master and develop
Oliver Bertuch (Mar 15 2023 at 18:41):This is a good idea anyway, as we avoid running the tests twice that way: once for the PR, once for the push
Philip Durbin 🚀 (Mar 15 2023 at 18:42):Just to clarify, we never push to master nor develop. These branches are protected. We do merge to them.
Oliver Bertuch (Mar 15 2023 at 18:43):What I want to know:
Oliver Bertuch (Mar 15 2023 at 18:44):@Philip Durbin @Guillermo Portas opinions plz?!
Oliver Bertuch (Mar 15 2023 at 18:48):Oh and 3. do we need some kind of report comment what the name of the generated image is, so one does not need to dig through logs?
Oliver Bertuch (Mar 15 2023 at 18:51):Philip Durbin said:
Just to clarify, we never push to master nor develop. These branches are protected. We do merge to them.
Absolutely - but when a PR is merged for these branches, this triggers a push event. So this is important to ship the stable/unstable images
Oliver Bertuch (Mar 15 2023 at 19:32):@Philip Durbin @Guillermo Portas WDYT? https://github.com/poikilotherm/test-image-push-flows/pull/4#issuecomment-1470702800
Guillermo Portas (Mar 15 2023 at 20:05):I would remove PR images once PRs are closed or merged. I would update the develop and master tags once anything gets merged into them.
Philip Durbin 🚀 (Mar 15 2023 at 20:06):What he said. I agree. :happy:
Philip Durbin 🚀 (Mar 15 2023 at 20:06):What does it mean to remove a tag? Does that delete the image?
Guillermo Portas (Mar 15 2023 at 20:08):It should delete that particular version of the image and make it no more accesible for pulling
Guillermo Portas (Mar 15 2023 at 20:10):So the decision is that we are only going to use GitHub registry, not Docker Hub. Correct me if I am wrong please!
Philip Durbin 🚀 (Mar 15 2023 at 20:16):It sounds @Oliver Bertuch might have three open questions. Let's please take them one at a time. :happy:
Philip Durbin 🚀 (Mar 15 2023 at 20:17):Oh and by the way, @Oliver Bertuch I just tried to get this PR into the sprint (we were in a kickoff meeting) but we have other work to do. Sorry! :sweat_smile:
Oliver Bertuch (Mar 15 2023 at 20:21):Philip Durbin said:
Oh and by the way, Oliver Bertuch I just tried to get this PR into the sprint (we were in a kickoff meeting) but we have other work to do. Sorry! :sweat_smile:
Meh. :melting_face: Containers still are not important.
Oliver Bertuch (Mar 15 2023 at 20:21):Philip Durbin said:
What does it mean to remove a tag? Does that delete the image?
That kind of depends on the registry policies
Oliver Bertuch (Mar 15 2023 at 20:23):Docker is much more strict with this than GHCR.
Oliver Bertuch (Mar 15 2023 at 20:23):On the other hand, usually GHCR storage isn't free
Oliver Bertuch (Mar 15 2023 at 20:24):It is free for public things on GHCR though
Oliver Bertuch (Mar 15 2023 at 20:24):So maybe we could use sth. like https://github.com/marketplace/actions/container-retention-policy to cleanup once in a while
Philip Durbin 🚀 (Mar 15 2023 at 20:25):Delete old images. Sure. Makes sense.
Oliver Bertuch (Mar 15 2023 at 20:26):So question 3 is already solved - I added an automated commenting already
Philip Durbin 🚀 (Mar 15 2023 at 20:26):Good! What's the next question?
Oliver Bertuch (Mar 15 2023 at 20:26):What remains is question 1: be more strict with building?
Oliver Bertuch (Mar 15 2023 at 20:27):Should we have some filter?
Oliver Bertuch (Mar 15 2023 at 20:27):Filter can basically be anything
Oliver Bertuch (Mar 15 2023 at 20:27):I mean maybe it's just fine to push images for all of it
Philip Durbin 🚀 (Mar 15 2023 at 20:27):What are the risks of simply pushing everything?
Oliver Bertuch (Mar 15 2023 at 20:27):It would help with testing
Oliver Bertuch (Mar 15 2023 at 20:28):As we don't pay a dime for our public images, I don't see much risk
Oliver Bertuch (Mar 15 2023 at 20:28):Of course CPU cycles might be a thing
Philip Durbin 🚀 (Mar 15 2023 at 20:29):Maybe we can leave a comment in some code or in a doc that we can start filtering if need be and link to how.
Oliver Bertuch (Mar 15 2023 at 20:29):RIght.
Oliver Bertuch (Mar 15 2023 at 20:30):I think I'm quite fond of how this works now in the pet project
Philip Durbin 🚀 (Mar 15 2023 at 20:31):Looks like you're building an image with Java in it.
Oliver Bertuch (Mar 15 2023 at 20:32):Yeah nothing fancy. Just some stupid stuff. Doesn't matter what is inside, the hard part was getting the flows right
Philip Durbin 🚀 (Mar 15 2023 at 20:35):Sure. Was there a last question? I've lost track. :sweat_smile: Thanks for hacking on all this! It's great! 
Oliver Bertuch (Mar 15 2023 at 20:36):I think I'm good
Oliver Bertuch (Mar 15 2023 at 20:37):I wonder if I should put something into a Dataverse PR from this...
Oliver Bertuch (Mar 15 2023 at 20:37):Given that even the most basic thing isn't even prioritized
Philip Durbin 🚀 (Mar 15 2023 at 20:39):Oh, we have a plan but you're not sure if it's worth coding it up? Because it isn't a priority?
Oliver Bertuch (Mar 15 2023 at 20:39):Basically this stuff is done
Oliver Bertuch (Mar 15 2023 at 20:39):It works as expected
Oliver Bertuch (Mar 15 2023 at 20:39):Has all the features to make people happy
Oliver Bertuch (Mar 15 2023 at 20:40):People can get going with this (at least for the application image)
Philip Durbin 🚀 (Mar 15 2023 at 20:40):Let's at least put an estimate on it, please.
Oliver Bertuch (Mar 15 2023 at 20:40):Size: 3
Oliver Bertuch (Mar 15 2023 at 20:40):Because I already did all the coding
Oliver Bertuch (Mar 15 2023 at 20:41):And for testing it's better to use the pet project. Turnaround times in the main project are gonna be tremendously larger
Oliver Bertuch (Mar 15 2023 at 20:41):I dunno if someone wants to write some docs how the pipeline works. Might be neat
Oliver Bertuch (Mar 15 2023 at 20:42):Haha even the scheduled execution worked like a champ
Oliver Bertuch (Mar 15 2023 at 20:42):https://github.com/poikilotherm/test-image-push-flows/actions/runs/4430739909
Oliver Bertuch (Mar 15 2023 at 20:43):It build the base image anew, deployed it to Docker Hub and then rebuilt the app image on top of the new base
Philip Durbin 🚀 (Mar 15 2023 at 20:43):Thanks. I gave it a 3 and left a comment: https://github.com/IQSS/dataverse/pull/9447#issuecomment-1470812282
Oliver Bertuch (Mar 15 2023 at 20:46):Thx. I added a comment about the pet project and the next step to get this into the PR
Philip Durbin 🚀 (Mar 15 2023 at 20:46):I think this is a better URL for the clean diff: https://github.com/IQSS/dataverse/compare/9434-app-container...9444-push-images . If you don't mind, I'll update the description. It's tiny! But this only pushes the base image, right? Not the app image.
Oliver Bertuch (Mar 15 2023 at 20:47):Yeah, these were some initial fixes of stuff.
Oliver Bertuch (Mar 15 2023 at 20:47):There's more in the pet project.
Oliver Bertuch (Mar 15 2023 at 20:48):I probably shouldn't have done it in the pet project. I won't get as much commit counts now!
Philip Durbin 🚀 (Mar 15 2023 at 20:48):So... what does it take to move this PR from draft to ready? Start pushing the base image?
Oliver Bertuch (Mar 15 2023 at 20:49):The base image will already be pushed. The addition in the PR was only about _scheduled_ pushes and a fix for getting a push to the stable tag when in main/master
Oliver Bertuch (Mar 15 2023 at 20:49):Philip Durbin said:
So... what does it take to move this PR from draft to ready? Start pushing the base image?
https://github.com/IQSS/dataverse/pull/9447#issuecomment-1470815116 :see_no_evil:
Philip Durbin 🚀 (Mar 15 2023 at 20:50):Oh, I was clearly confused. This is about stable. Got it.
Oliver Bertuch (Mar 15 2023 at 20:50):"ci(ct): fix pushing the correct tag (stable) for master branch" :-D
Philip Durbin 🚀 (Mar 15 2023 at 20:50):Ok, and copy workflows, add docs, etc. I think I get it.
Oliver Bertuch (Mar 15 2023 at 20:51):The hard part is done.
Philip Durbin 🚀 (Mar 15 2023 at 20:52):Right. And the details are in the pet project.
Oliver Bertuch (Mar 15 2023 at 20:52):-details +solutions
Oliver Bertuch (Mar 15 2023 at 20:52):The details are the somewhat missing docs in a page in the guide
Philip Durbin 🚀 (Mar 15 2023 at 20:53):Ok. Still a 3. :happy:
Oliver Bertuch (Mar 20 2023 at 19:52):So, with #9434 resolved, one next step is "do some push ups". (Pun intended)
Philip Durbin 🚀 (Mar 21 2023 at 18:50):I did give this a 3 and I just moved it to the top of the Dataverse Team column in the backlog.
Instead of "push images" can we give it a more specific title? "push dev app images to registries" maybe?
Oliver Bertuch (Apr 24 2023 at 06:48):Morning @Philip Durbin - I am adding the necessary workflow scripts to push the images (so we can fulfill Milestone B). Would you be so kind to create two secrets in IQSS/dataverse for me? (GHCR_USERNAME & GHCR_TOKEN)
Oliver Bertuch (Apr 24 2023 at 12:21):Thank you!
Oliver Bertuch (Apr 24 2023 at 14:09):What should we put into the README for gdcc/dataverse that will be pushed to Docker Hub?
Oliver Bertuch (Apr 24 2023 at 14:09):Suggestions welcome!
Oliver Bertuch (Apr 24 2023 at 14:32):OK here's a draft. https://github.com/IQSS/dataverse/pull/9447/files#diff-774e1f5fbbba6ff5f99dbdc3457c545bb945a88402b69080516ecca7fcd4fdce
Philip Durbin 🚀 (Apr 24 2023 at 15:40):Looks good!
Philip Durbin 🚀 (Apr 24 2023 at 15:40):And I see the PR is no longer draft!
Oliver Bertuch (Apr 24 2023 at 15:46):Yes it isn't. Might need polishing by review, otherwise probably good to go
Oliver Bertuch (Apr 24 2023 at 15:46):And some front-end dev should test pulling from GHCR. Maybe @Guillermo Portas is up for it?
Philip Durbin 🚀 (Apr 24 2023 at 15:48):@Guillermo Portas was JUST talking about the need for a containerized dev env: https://github.com/IQSS/dataverse-frontend/issues/65
I suspect he'll be happy to see this pull request is no longer a draft! :happy:
Oliver Bertuch (Apr 24 2023 at 15:49):There is a low volume but constant flow of output here
Oliver Bertuch (Apr 24 2023 at 15:49):I would be glad to know if that crazy idea from my head is now actually of any use
Oliver Bertuch (Apr 24 2023 at 15:50):Limited so far, of course. But a step in the hopefully right direction
Philip Durbin 🚀 (Apr 25 2023 at 11:27):I merged https://github.com/IQSS/dataverse/pull/9447
Philip Durbin 🚀 (Apr 25 2023 at 11:27):Now we have https://hub.docker.com/r/gdcc/dataverse
Philip Durbin 🚀 (Apr 25 2023 at 11:27):Thanks, @Oliver Bertuch !
Philip Durbin 🚀 (Apr 25 2023 at 11:27):@Guillermo Portas check this out ^^
Guillermo Portas (Apr 25 2023 at 13:54):Awesome! I will be happy to test it when working in the development environment for frontend developers
Oliver Bertuch (Apr 25 2023 at 15:46):Interesting first find already: the PR from ErykKul did not go through with pushing the PR Image because his workflow run does not have permission to access the credentials. Let's see if it works property for IQSS folks
Philip Durbin 🚀 (Apr 25 2023 at 20:48):This one was pushed: https://github.com/orgs/gdcc/packages/container/dataverse/88346793?tag=9361-storage-quotas
Via https://github.com/IQSS/dataverse/pull/9409#issuecomment-1522340389
Oliver Bertuch (Apr 26 2023 at 07:52):Yes of course - that PR is based on a branch living at IQSS/dataverse and thus has access to the secrets. PRs with branches in forks don't have access to the secrets, so they won't be pushing images.
Oliver Bertuch (Apr 26 2023 at 07:54):Probably the only way to resolve that is to push these images not to GDCC but IQSS as packages, because then the GITHUB_TOKEN should be enabled to write
Philip Durbin 🚀 (Apr 26 2023 at 11:03):Sounds fine to me. I'm not aware of any downsides.
Oliver Bertuch (Apr 26 2023 at 12:00):I might have another idea. We could try using comments for these cases.
Oliver Bertuch (Apr 26 2023 at 12:01):We can trigger actions on comments. So some core dev could trigger pushing the image by leaving a certain comment
Oliver Bertuch (Apr 26 2023 at 12:02):This _should_ run the workflow with all the access to secrets...
Philip Durbin 🚀 (Apr 26 2023 at 12:45):Interesting. Sure, worth a shot.
Philip Durbin 🚀 (Apr 26 2023 at 14:28):I just added "push to registry" (this topic) to the agenda for tomorrow's meeting: https://docs.google.com/document/d/1Hz47lLjE9h1-YE5zD2wu4tT1ObB6vB6Nr3m16pQ4LF4/edit?usp=sharing
Oliver Bertuch (Apr 28 2023 at 07:09):@Guillermo Portas I just added a parameter to set the tag in the command comment. It will still default to branch name, but you can override via named parameter. Here's the example: https://github.com/poikilotherm/test-image-push-flows/pull/8#issuecomment-1527054755
Oliver Bertuch (Apr 28 2023 at 10:15):Note to self: make the job status visible by adding it as a check with https://github.com/marketplace/actions/github-checks
Oliver Bertuch (May 02 2023 at 08:29):@Philip Durbin may I require your assistance this evening during tech hour?
Philip Durbin 🚀 (May 02 2023 at 11:01):Sure, what can I do for you?
Oliver Bertuch (May 02 2023 at 11:27):Would you be so kind to create a fork of https://github.com/poikilotherm/test-image-push-flows, edit something so you can create a pull request this evening?
Oliver Bertuch (May 02 2023 at 11:27):Just so we have a proper demo of what happens...
Philip Durbin 🚀 (May 04 2023 at 16:37):Whoops, I guess I missed this, sorry.
I just started a fresh topic for /push-images: https://dataverse.zulipchat.com/#narrow/stream/375812-containers/topic/.2Fpush-images
Notification Bot (May 08 2023 at 06:17):Oliver Bertuch has marked this topic as resolved.
Last updated: Oct 30 2025 at 05:14 UTC