Today in the containerization meeting we discussed the need for images to be tagged with version numbers such as 5.14, 6.0, etc.
We looked at https://hub.docker.com/_/solr/tags as an example of what we want.
Screen-Shot-2023-06-15-at-3.09.03-PM.png
Someone pointed out that "latest" and "9.2.1" have the same digest. This is normal and expected.
It's nice to point to 9.2.1 because you know it won't change under your feet. This is not true of the "latest" tag.
I couldn't find anything about this in our roadmap. I'd suggest that we create a GitHub issue for it when we're ready.
A big question I have is, when we release 5.14, should we push a 5.14 tag to DockerHub? I don't think we're ready to do this honestly. It'll take some work to prepare for this.
Philip Durbin said:
A big question I have is, when we release 5.14, should we push a 5.14 tag to DockerHub? I don't think we're ready to do this honestly. It'll take some work to prepare for this.
Out of curiousity, what obstacles do you see here?
Problem with version tags is about getting outdated.
Releasing updated images for old versions is creating a lot of effort
The problem here is not just technical.
It's a thing about commitment
IQSS is not able to support usage of Docker images for production (yet?)
So this is sth that the community needs to look into
And we do not have a team of platform engineers yet that is working together here
Maybe, when we have a more sophisticated testing pipeline, we could create more supported versions than latest main and develop...
And it's not done with just the app image getting tags. We would need to do tagged base images as well, as we would need to have certain versions of Payara around etc
I'm not really that sure it's worth it.
If you are concerned about new image versions, go use the sha256 to pin down the docker image version.
That's good practice anyway
Oh and not to forget: we'd need versions of configbaker, too...
If the community says we don't care about security, just give us frozen in time images with potential sec risks, sure, let's do it.
Yeah, alright, that makes perfect sense; maintainability of the images is a factor and if there aren't the engineering resources to drive it forwardd (consistently and securely) then it's not a good hill to die on.
I guess I'm a little confused. We COULD push a 5.14 image and never update it. It would be nice and reproducible. However, a log4j or whatever vulnerability could be in there. And that's bad, obviously, security-wise. But if you push a new 5.14 doesn't that defeat the purpose of the image being stable?
I would say ordinarily, you would push a 5.14.1 (same as you would with "regular" software). However, I can see the engineering effort of testing/maintaining/distributing that on two separate channels can be exhausting. I would say delaying a (public, versioned) release has the benefits of not creating expectations in that manner as well; if you haven't released it, it hasn't broken in production (yet).
Hmm. Maybe we should touch on this in the containerization meeting in a couple hours.
I did touch on it but only briefly. I linked back to this topic.
I'm silly that I keep missing those. I'm going to plan that timeslot regular to inform people (especially Management) they shouldn't be Muggling about during that period.
Heh. No worries. We can talk about it tomorrow if you like!
We had a fairly robust discussion about tagging in the middle of yesterday's meeting: https://harvard.zoom.us/rec/share/vt5PT7OH-Jw3w_orzEhfH1LVCb_cbpNZj3zbz9rsMmqotqLe0aDutti2CoKeJ6MF.Tqs7XJ648HnCvoj7
I there a change to tag dataverse releases with something like alpha:v6.1 and share them via dockerhub?
We discussed this briefly yesterday. Here's the recording: https://harvard.zoom.us/rec/share/-e5CHUCBg7STDie4RF7dFx-YHB74pYQewg6hzKUAuNFBTrD_S7UvH0WrlvZ95erw.eVB72Eyc16Y_kcHE
Specifically we were talking about #containers > change version scheme base image? but it's all related.
Thanks for the link, I'll watch the recording: )
Sure. I think we mentioned you a couple times. :sweat_smile:
And it's pretty short. Don't worry. :grinning:
Yeah I heard my name a few times ;)
only good things :grinning:
IHMO: Build matrixes, with different build options, would be nice but the added complex is not worth the hassle. The "official" release only includes a single war file and no different compiled versions. We are all happy with that... For docker images, a single tag for each release would be sufficient for the beginning. The remainder images are here and can be used to test against a PR. If we really want access to each specific commit, then we need to push and tag each commit. Most likely the majority of those images won't be used and we need to define a retention cycle unless we want to use all the storage ;). Therefore, I believe a single tag for each release is the best option.
I just wanted to mention that we had a fairly robust discussion about tags yesterday in the container meeting: https://harvard.zoom.us/rec/share/rDo_V6heO9vo6UDIVhm0mnnsdTDXqA4lNX72lCjhGWrmOEpZttM13CsBceAFXkvr.SgokYNw-AoOACtSm
Notes are a bit scattered across the regular notes and a doc I'm using for #containers > tutorial for demo or eval #10238 .
Better to watch the video, I think. :grinning:
@Oliver Bertuch I see you marked the other thread resolved, which makes sense. Should we continue using this one?
I just requested a review from you for this PR: bump version to 6.4 #10871
I added the extra change we need for containers (and made noise about it at standup this morning).
This thread is linked from https://guides.dataverse.org/en/6.3/container/running/production.html by the way, so it might be nice to keep it open.
Sounds good to me :saluting_face:
@Oliver Bertuch I think we have only one more PR we want to merge before releasing 6.4. Any news on #10618 (app image tags)? Should we hold the train?!? :train:
No please carry on. I will not be able to work on this before next week.
Fair enough! Thanks!
We'll release a 6.4 image once we have the workflow going
Yeah, we'll build on it.
Just like we did with 6.1-6.3 now :smile:
Yeah. Anyway, we just kicked off a new sprint. I dragged #10618 into it. Hopefully we'll get to it in the next two weeks. If not, we can kick it back. :grinning:
SGTM! :sound: :+1: /me
I gave it a tiny size because last time you did all the work but please let me know if I can help at all.
When discussing making the 6.4 release today I explained toggling between these:
<base.image.version>${parsedVersion.majorVersion}.${parsedVersion.nextMinorVersion}</base.image.version>
<!-- <base.image.version>${revision}</base.image.version> -->
@Steven Winship asked if we could automate this as part of our release process. We all agreed it's a good idea, but our release process is highly manual right now. :grimacing:
My money would be on sth like https://www.mojohaus.org/versions/versions-maven-plugin/set-property-mojo.html
Hmm, should we start a topic for this?
You kind of did in #dev > From Jenkins to GHA to Dagger? , too. As it's not just containers, maybe talk about it in a #dev topic?
Yeah, I'm thinking #dev
So can I help in any way with this issue? Tags for application container imagesΒ #10618
I assigned it to myself at the start of the sprint (which is now almost over) but I haven't done anything with it. :sweat_smile:
@Oliver Bertuch do you figure we'll talk about tagging during today's call? I don't have much on the agenda.
So I'm looking into app container image versions right now.
I'm wondering if for the time being I should actually touch any of the usual app image pushing and stuff
I could also just use the maintenance job and let that take care of the app image stuff on its own. Maybe that would reduce complexity. (It's already VERY complex)
Any thoughts?
Yay!
I'm not sure I'm following. Whatever you feel is best!
Happy to do a zoom or whatever if it helps!
Yeah, that might be a good idea.
Should I light a fire?
I just did: https://fz-juelich-de.zoom.us/j/62913876486?pwd=E0IcMtxb7ZufoDYp9rLNbpDDxGCaKS.1
container working group, assemble!
(but I need to find a room) :sweat_smile:
@Oliver Bertuch great chat, thanks for working on #10618! Please let me know if we should move it from "on hold" to "in progress"!
Yes, please move it :slight_smile:
Actually now that there's a PR (#11477) I moved the PR to "in progress" (and mentioned it at standup) and removed the issue from the project (which is our way). :smile:
@Don Sizemore any ideas why your guides action wouldn't install the dependency? Worked on my machine... https://github.com/IQSS/dataverse/actions/runs/14927388354/job/41935232979
@Oliver Bertuch I'm afraid not, off the top of my head. The only reason we have that fork is: uses -W instead of --keep-going
I see the .cache permissions error, but I've done nothing special to the upstream sphinx container.
I'm suspecting that the base image (sphinx:3.5.4) has an old Python version that might not be compatible.
It's using 3.9 as seen in the image. The package requires 3.10 (as seen here)
@Don Sizemore can this be upgraded? Simply switching to sphinx:7.4.0 would probably help :smiley:
Should we open a new PR to see if it's happening everywhere?
No need - it's just for me because I installed another extension
We are installing Sphinx 7 anyway by installing it via pip/requirements.txt, so it would probably make sense to update.
Let's open an issue and yes sounds like upgrading Sphinx is the way to go.
I can't open an issue - the repo https://github.com/uncch-rdmc/sphinx-action has them deactivated.
I could create a PR though :smiling_face_with_hearts:
@Philip Durbin βοΈ this isn't a good look :cry:
image.png
Do you think we should enable the Renovatebot to update our container deps/packages/...?
@Oliver Bertuch I had thought you might open an issue in the IQSS repo (and potentially do away with the uncch-rdmc fork altogether?)
Well we could pull the action into the IQSS main repo and make it our own action. Is that what you're suggesting? Add it to https://github.com/IQSS/dataverse/tree/develop/.github/actions and use from there?
@Oliver Bertuch that uncch-rdmc fork was a stop-gap; it would be preferable for it to go away.
@Don Sizemore I see!
@Oliver Bertuch or, if we still need it, that's fine as well. just an extra hop, is all.
Well I can create an issue. Wouldn't try to rope the moving into this PR, but would like to get it unblocked. Can we still update the current bandaid before we rip it off?
sure thing. speaking of band-aids, I still remember the pain of upgrading to our currently-used version of Sphinx.
@Oliver Bertuch yes to renovatebot, please!
in the short term, I've put testing with sphinx:7.4.0 on my never-ending to-do list.
What would you needed to be tested? Some other workflows using this action?
Philip Durbin βοΈ said:
Oliver Bertuch yes to renovatebot, please!
Great. I'll make it do weekends, group by image and only work on ct stuff, no Java.
We can extend from there...
Way more options than dependabot :cowboy:
Tried this out today. It works (see https://github.com/gdcc/wip-dataverse-base-image/issues/3#issue-3065663570), but there is a big BUT. Looks like in Alpine, they do not save older versions. So there is no reproducible builds, because once the version progresses on their side, there is no installing an older version...
So the only way to go about this is to stick to the current approach without versions and detect in the maintenance script if updates for them are available
Might be good to save this link, in case we need a Renovate Bot configuration again at some later time. https://github.com/gdcc/wip-dataverse-base-image/commit/53843cc0c4f0299435706a72af3e6ff980fa7a2d
Should we switch from Alpine? Is there some other image that saves older versions?
Not images, package ecosystems. Yeah, Ubuntu etc keep older versions for longer
I wasn't sure what to call it. :sweat_smile:
I really am thinking about using Ubuntu here as well. The images would be aligned and we can reuse the scripts we already have to detect the new package versions.
We'd need to install some stuff not from package managers, but that might not be that bad. We can use Renovate to keep 'em updated
No objection to switching to Ubuntu.
Haha I read up a little about Ubuntu's policy on this - same... They don't keep old packages, only very temporarily.
So much for reproducible builds! ![]()
Using Ubuntu means we blow up the image by ~100%. 239 MB -> 419MB
And I had to use hacks to work around stuff that is not packaged for Ubuntu :rolling_eyes:
Like install pipx via apt, then install pipx with pipx, delete apt version, install pipx with pipx globally just to install awscli without crazy python3-pip package dependencies in Ubuntu...
Otherwise it would have been ~300MB more... Crazy Ubuntu packaging....
Well, at least we're standardizing.
Yeah. Also probably not a bad idea to have python and a recent pipx around
Can be used for scripting
@Philip Durbin would you agree that we should keep the Alpine flavors for the past releases and not change those to Ubuntu for the sake of backward compatibility?
(So switch to Ubuntu going forward only)
This is only for configbaker, right? Does it matter much? As long as it works, right?
Yeah, that's what I figured.
Whatever is easier, I'd say. Whatever is less complicated.
As it means less patch files, sticking with Alpine for the moment seems less complex
for past releases
sure, sounds fine
@Don Sizemore I fixed Sphinx with https://github.com/IQSS/dataverse/pull/11477/commits/db86eb1a5cfbf896ae11f073891c0c1c14894f0e
BTW for the configbaker image I'm introducing a Trivy Scan in case there's no more recent upstream image. This way we can learn if there are sec vulns that can be fixed by updating (=rebuild)
For now, this will only look at OS packages, but we can extend that further.
Keeping it a bit simpler for now
KISS!
Might be helpful for the other images as well. We'll see.
@Oliver Bertuch wonderful! I'm sorry this languished on my to-do list.
No worries. I didn't make a very sophisticated thing of this, just reusing things already available. And who knows - maybe we publish the docs to GH pages or some place else using GH actions some day.
@Philip Durbin reason enough to upgrade the older images, too?
image.png
I mean, no objection from me. I appreciate all the work you're doing on these images!
Bah I'll leave it at that for now. We can still switch to a different policy at a later point
@Philip Durbin the PR is mostly done (missing a release snippet and docs in the devguide). Do you have spare cycles to take a look? It would be good to have the PR merged this week so I have images ready to go for next week... :wink:
I can walk you through as well :smile:
It looks huge, but isn't that complicated...
Sure! Want to do a quick call?
So we can do a Zoom now if you want and don't mind me eatin' :wink:
No camera anyway :smiley:
yes, no camera please :smile:
let me grab sonia's office
The ipu6 Intel cam doesn't work on Linux (yet), so no worries :smiley_cat:
https://fz-juelich-de.zoom-x.de/j/63874239919?pwd=NRA5yGhjPc35teaSCb1agMt5pHcIVK.1
Why do we already see tags like 6.6-noble at https://hub.docker.com/r/gdcc/dataverse/tags even though #11477 hasn't been merged yet? This is because of the existing maintenance script from #10827?
Probably was me during tests on my local machine... :melting_face:
Ok, no worries, thanks.
This doesn't make much sense to me, under "Upcoming":
Summary: Rolling tag, equivalent to unstable for current development cycle. Will roll over to the rolling production tag after a Dataverse release.
Do we even need "upcoming"? Who would use it?
Someone you wants to use the next release version in some evaluation but not get the changes from the next dev cycle
Because the upcoming versioned tag will be the next stable rolling tag
If someone wants a preview of 6.7, for example?
Yeah exactly
And then maybe stick with it
Some staging environment where the version gets rolled over to the production environment once satisfied and released
for staging
but wait, why not just use "unstable"?
Because that will always point to the latest dev snap
Let's have an example
please!
Say I'm testing 6.7
That's not released at the moment
ok
So it's the same as unstable
specifically, the tag is 6.7-noble right? but we'll call it 6.7 for short
But as I want to stick to a certain level of development, maybe eager for some feature to be in 6.7, I still use the version to make it explicit what I am doing - I'm interested in the 6.7 release, not following the dev cycle beyond it
Now 6.7 gets released
So unstable will now be the same as 6.8-noble
And I get the stable, maintained images sticking to 6.7
So I can make a PR from my staging branch for GitOps to production and there's 6.7 that's getting inserted
So I don't need to make a manual change from unstable to 6.7
I can definitely see someone using such a workflow
And yes, using 6.7-noble shortened to 6.7 for less typing ... lazy dev here
Does that make sense now?
Let me try to express this another way.
You decide on 6.7-noble as the tag you want. 6.7 hasn't been released yet. You merrily test away.
Eventually, 6.7 is released. At this point, the 6.7-noble tag you chose is the same tag that someone would use if they want the released version of 6.7. You don't have to do anything (well, maybe a docker pull). You're already on the tag you want.
Yeah exactly
Ok, interesting use case.
I'm hacking away on these docs. I'll have you review them. :smile:
It's actually pretty common in DevOps/GitOps to have expressive statements of what you want
declarative
Yeah
Sry
Poor choice of wording here
We probably can, at a later point, introduce a tag like 6-noble to further this
Which would always point to the latest stable minor release
You would need to docker pull, right? When 6.7 comes out?
Absolutely
ok
In case of Kubernetes, you'd use an ImagePullPolicy of Always
So when the pod is restarted, it gets the new image immediately
ha, now I want a |nextNextVersion| :crazy:
Ok, I just pushed. Please take a look.
There's a bit of duplication going on with all the "supported image tags" stuff.
But I'm not going to worry about that right now. I'd like to move this along.
Yeah, I don't fancy the dupes. But if someone goes to one of the pages, all of the info is right there, no need to go to another site.
So someone looking for specific information on one of the images can find it all in one place
I feel like 99% of people will only care about the dataverse image, the app image.
Also, if we change the release model in the future for the different pieces, it would be good to already have a proper szart
There's much truth in that 99%...
yeah, it's fine
anyway, please edit away those docs
They look fine
revert my ideas as needed :smile:
No no it's all good. Someone else should also get it, too, and you have a different context and perspective than me, so it's good to mix that into all.of this.
As I look at the code, is there anything in particular you have doubts about?
You mean something in the maintenance scripts that might fall apart when looking to closely?
just in general
I'm not sure I understand this:
"It needs to be decided if not running the Maven Unit Test workflow on push to master and not on new tags.
Otherwise, this is only touching container stuff, low risk for anyone else."
should we decide now?
what are we deciding? :smile:
In general my doubts are about the complexity of all of this
sure
But it is what it is... ![]()
yeah
Folks want versioned images, so let's do it
yes!
Wrt maven tests
I talked it up at standup today
Should we do a quick zoom?
bah, I need a room. one sec
I can also type it out
Painstakingly on my cell
:sweat_smile:
do you have a link?
Just a sec plz
https://fz-juelich-de.zoom-x.de/j/63096995209?pwd=I1rVyV0SqnesiYRyRT2jRIqMJVJnV8.1
Last updated: Oct 30 2025 at 05:14 UTC