Notification Bot (Jul 20 2026 at 15:56): Notification Bot (Jul 20 2026 at 15:56):This topic was moved here from #containers > containerized-testing-suite by Philip Durbin 🚀.
Philip Durbin 🚀 (Aug 27 2026 at 13:59):@Ash Manda I'm not sure if you want me to warm up this topic every time a JSF test fails or if I should create a new one but can you please take a look at https://github.com/IQSS/dataverse/pull/12614/changes#r3872417379 from a PR by @Florian Fritze?
Ash Manda (Aug 27 2026 at 14:31):I'll take a look at the videos
Philip Durbin 🚀 (Aug 27 2026 at 14:32):@Ash Manda Thank you!! 
Ash Manda (Aug 27 2026 at 14:33):This is flakiness related to this issue: https://github.com/gdcc/dataverse-jsf-tests/issues/1
Ash Manda (Aug 27 2026 at 14:35):8062dd3ae4b393ec9666e53896eca3420d66087d.webm
Philip Durbin 🚀 (Aug 27 2026 at 14:37):Ok, so you think this is a case of flakiness? Not because a subfield was added?
Ash Manda (Aug 27 2026 at 14:37):This has been particularly difficult for me to diagnose because I have never seen it occur on my computer no matter how many times I run the test. But, for now I think any failure related to dataset actions can be safely skipped including this one
Ash Manda (Aug 27 2026 at 14:37):Philip Durbin 🚀 said:
Ok, so you think this is a case of flakiness? Not because a subfield was added?
Nope, this is 100% flakiness.
Ash Manda (Aug 27 2026 at 14:47):I am working on a hotfix right now, I'll keep y'all updated.
Philip Durbin 🚀 (Aug 27 2026 at 14:48):Amazing! Thanks! :heart:
Ash Manda (Aug 27 2026 at 14:58):I am going to submit a fix directly to gdcc/jsf tests, so nothing has to be merged into upstream
Philip Durbin 🚀 (Aug 27 2026 at 14:59):Sounds perfect. You can mark the PR to fix https://github.com/gdcc/dataverse-jsf-tests/issues/1 I assume.
Ash Manda (Aug 27 2026 at 15:00):Could you re-run the tests now? Dataset actions should be more reliable now.
Ash Manda (Aug 27 2026 at 15:02):Apologies for the noise this flaky test has caused upstream. The failure in 13-dataset-actions.spec.ts only surfaces in GitHub Actions. Locally the timing and hardware is forgiving enough that it passes consistently, which made it hard to catch and diagnose earlier.
The root cause seems to be a race condition: after saving metadata, the PrimeFaces DataTable AJAX cycle hadn't fully settled before we were interacting with the select-all checkbox. I've merged a PR with a fix that waits for network idle before the interaction and targets the correct inner PrimeFaces element. Should be stable going forward.
Philip Durbin 🚀 (Aug 27 2026 at 15:30):Sure! Re-running! https://github.com/IQSS/dataverse/pull/12614#discussion_r3873240802
Ash Manda (Aug 27 2026 at 15:52):
Yay!
Philip Durbin 🚀 (Aug 27 2026 at 16:13):Passing! Thanks again, @Ash Manda !
Ash Manda (Aug 27 2026 at 16:14):Of course! Sorry I hadn't fixed this before, I just wasn't sure on what to fix
Ash Manda (Sep 09 2026 at 19:29):Hello! Heads up to the team. Over the past few months I have written about a dozen or so more tests and I even used some tests over from the QDR repository that I got access to, thanks to Jim. I am currently testing the stability of the new tests within GitHub Actions. Assuming they stay stable enough throughout this week, there will be a merge into gdcc/dataverse-jsf-tests next week and you will be seeing longer runtimes and more results for the JSF workflow in the main upstream dataverse repo.
I realize that IQSS is in a code-freeze right now, and this should be a relatively low risk time for the merge, but there's always a chance of seeing increased failures with tests. Please do let me know if you want me to postpone this or discuss this or anything.
Thank you!
Philip Durbin 🚀 (Sep 09 2026 at 19:32):As I mention whenever I can, I think we should move the JSF tests into the main repo, next to the JSF code.
I think the only downside is that installations might have their own tests. But they already fork the main repo so I think it would be more of the same.
Ash Manda (Sep 09 2026 at 19:53):I'm more than happy with that decision if the team is okay with it. From my side not much is going to change. I just want to make sure that this suite doesn't become a maintenance burden for the team
Philip Durbin 🚀 (Sep 09 2026 at 19:56):It's a gift! Free code! :tada:
Want me to make an issue about moving the code over?
Ash Manda (Sep 09 2026 at 20:04):Let me clear it with Don once
Oliver Bertuch (Sep 10 2026 at 14:59):@Ash Manda thanks a ton for these tests . While I was waiting for the API tests to complete, the JSF test failures already uncovered a problem in my latest PR. I was able to unravel the root cause from digging into the report and the logs. This is a huge step forward!
Oliver Bertuch (Sep 10 2026 at 15:08):Let me be bold and suggest some improvements: it would be even much better if these tests would...
... be part of the regular chain (unit, integration, API + JSF).
... be easier to correlate. A failing frontend test and what may have gone wrong in the backend need to be put together manually. I understand this is a deep deep rabbit hole, but maybe it's worth looking into options how to make OpenTelemetry spans from Playwright propagate to the backend so we have end-to-end snapshots?
... have a nicer web UI access for their reports. Downloading the Playwright reports is slooooow and the ZIP is largeish. (1MB/s throughput, 118MB zipfile). Can we upload these HTML reports somewhere else?
Ash Manda (Sep 10 2026 at 15:31):Hi Oliver. Thank you so much for the suggestions! I have thought about all of these things at one point or another, and all of them are on my roadmap. My biggest worry with having the image be built exactly once and having the three tests run in line is state pollution. I was wondering if you had any thoughts on that?
Ash Manda (Sep 10 2026 at 15:32):That and if say an API test fails, we probably don't want it to block the pipeline and still have it run the JSF tests anyways. So I felt in the beginning it made a lot more sense to run them separately.
Oliver Bertuch (Sep 10 2026 at 15:32):What kind of state pollution are you anticipating?
Ash Manda (Sep 10 2026 at 15:34):results from first tests being included while running the last tests. Like you know not depending on what happens in previous tests, if that makes sense. We faced some issues with this before I believe in the API tests where the count in one of the tests would go off. With so many tests run line by line, I might unintentionally start building my tests in a way that they would depend on previous ones.
Oliver Bertuch (Sep 10 2026 at 15:35):Ash Manda said:
That and if say an API test fails, we probably don't want it to block the pipeline and still have it run the JSF tests anyways. So I felt in the beginning it made a lot more sense to run them separately.
Yes of course! Maybe this was ambigous above! From where I am standing, integration, API and JSF tests should be done in parallel, once compilation, unit tests and packaging is done. (integration tests don't need packaging of containers, so they can start early)
Ash Manda (Sep 10 2026 at 15:36):There's probably a way I can get the unit, integration, and JSF tests all run in parallel and produce the reports at the same time. Let me stew on this a bit.
Oliver Bertuch (Sep 10 2026 at 15:36):My main point about an "overarching thing" was that once all these tests are done, we need to collate the different reports, merge them (in case of coverage) and do some more things like SonarCloud Analysis, etc
Oliver Bertuch (Sep 10 2026 at 15:36):Github action's jobs are meant to be run in parallel. I'm already doing this in some of the existing pipelines.
Ash Manda (Sep 10 2026 at 15:36):I am not sure where we could upload the Playwright reports though. S3 maybe? But you pointed out the issue, these reports can't be statically served
Ash Manda (Sep 10 2026 at 15:37):so maybe some kind of lambda?
Oliver Bertuch (Sep 10 2026 at 15:37):Jobs are only ever run sequentially once you define dependencies between them. Of course a jobs step are always sequential.
Ash Manda (Sep 10 2026 at 15:40):How does Sonar ingest its coverage metrics? Like how can I help use these test reports ingest into Sonar?
Oliver Bertuch (Sep 10 2026 at 15:40):Ash Manda said:
results from first tests being included while running the last tests. Like you know not depending on what happens in previous tests, if that makes sense. We faced some issues with this before I believe in the API tests where the count in one of the tests would go off. With so many tests run line by line, I might unintentionally start building my tests in a way that they would depend on previous ones.
But these tests would still run within their own realm, right? I agree that we must take care of running each test in a clean environment. With the setup as is I don't see much danger to we'd be able to fall into that trap though. (Ephemeral, separated deployment using compose). This might change if we ever think about moving to use JUnit+Testcontainers to control our pipeline. But we're not there (yet)...?!
Ash Manda (Sep 10 2026 at 15:41):Even if we run the jobs in parallel, I was under the assumption that GitHub would reuse the same build.
Oliver Bertuch (Sep 10 2026 at 15:42):Ash Manda said:
How does Sonar ingest its coverage metrics? Like how can I help use these test reports ingest into Sonar?
The sonarcloud job is defined here: https://github.com/IQSS/dataverse/blob/develop/.github/workflows/maven_unit_test.yml#L184-L220
It consumes the JaCoCo reports we put in place by downloading the artifacts from previous jobs. We configure the location in the POM.
Oliver Bertuch (Sep 10 2026 at 15:44):Ash Manda said:
Even if we run the jobs in parallel, I was under the assumption that GitHub would reuse the same build.
No, that is not possible. Each job (which can be parallelized) runs in an ephemeral container, thus must either rebuild or download artifacts. This is very different to what you get from GitLab CI or Jenkins by default (depening on setup/flavor/runner type).
Ash Manda (Sep 10 2026 at 15:46):That's really cool, I didn't know that
Oliver Bertuch (Sep 10 2026 at 15:54):Pipeline Engineering is a mess. :wink:
Ash Manda (Sep 10 2026 at 15:54):If I could make it 3% better than it was yesterday, that's a big win to me! Those Gremlins never go away!
Oliver Bertuch (Sep 10 2026 at 15:55):Kaizen is a smart way to live and work. :slight_smile:
Philip Durbin 🚀 (Sep 10 2026 at 15:58):Ash Manda said:
Let me clear it with Don once
@Ash Manda any news? :smile:
Ash Manda (Sep 10 2026 at 15:59):Don wanted to know if it could be a submodule. And I was like it could, but its gonna change y'alls workflows going from git pull to git submodule add and whatever not. The final decision is his, I'll let him stew on it :smile:
Ash Manda (Sep 10 2026 at 16:00):Worst case, you move it into the main repo, and I set up a .sh script or a chron job to maintain a repository with just the tests so that others could extend it (which is what I do for our cloak setup anyways) so we gotchu covered?
Philip Durbin 🚀 (Sep 10 2026 at 16:00):We've been able to avoid submodules so far.
Ash Manda (Sep 10 2026 at 16:01):But I was told that Cypress was preferred right from the very beginning, are you sure you want a Playwright suite for JSF and maintain a Cypress suite for the frontend
Ash Manda (Sep 10 2026 at 16:01):Like I said, I don't want to create unnecessary maintenance burdens
Philip Durbin 🚀 (Sep 10 2026 at 16:02):Well, let me ask you a related question. Should developers be writing JSF tests when they change the JSF code?
Ash Manda (Sep 10 2026 at 16:04):In Theory yes, but its going to be a burden. Frontend tests are a lot of clicking around and waiting to see what happens. It takes a lot of time
Philip Durbin 🚀 (Sep 10 2026 at 16:04):Sure, so maybe it's optional.
Philip Durbin 🚀 (Sep 10 2026 at 16:04):but
Philip Durbin 🚀 (Sep 10 2026 at 16:04):The tools could be at hand, in the same repo.
Philip Durbin 🚀 (Sep 10 2026 at 16:05):Right now there are no tools. No patterns to follow.
Ash Manda (Sep 10 2026 at 16:05):to be frank, you could just go ahead and do it. Just copy the tests over. And the separate repo can still be there.
Ash Manda (Sep 10 2026 at 16:05):But there's a new batch of tests coming over, and rn they're sitting in a PR on the gdcc repo. So maybe a week or two?
Philip Durbin 🚀 (Sep 10 2026 at 16:05):Ha, sure. I'm just hoping for some help with the GitHub Actions. To get it working.
Ash Manda (Sep 10 2026 at 16:06):of course! It's going to be a minor change to the Actions. Just cd instead of git clone
Philip Durbin 🚀 (Sep 10 2026 at 16:06):Plus, I'd be happy for you to get credit for the commits. It's your code!
Philip Durbin 🚀 (Sep 10 2026 at 16:07):And we can git blame you. :crazy:
Ash Manda (Sep 10 2026 at 16:07):I'm always having fun, until it becomes my responsibility
Ash Manda (Sep 10 2026 at 16:07):just kidding
Ash Manda (Sep 10 2026 at 16:07):technically its the community's code now
Philip Durbin 🚀 (Sep 10 2026 at 16:07):definitely
Philip Durbin 🚀 (Sep 10 2026 at 16:08):a group effort
Ash Manda (Sep 10 2026 at 16:08):Ash Manda said:
But there's a new batch of tests coming over, and rn they're sitting in a PR on the gdcc repo. So maybe a week or two?
Is this timeline okay?
Ash Manda (Sep 10 2026 at 16:08):Then I will help with moving them over and setting up the Actions
Philip Durbin 🚀 (Sep 10 2026 at 16:08):Sure! I don't want to mess with the JSF tests in the middle of a release anyway! :sweat_smile:
Ash Manda (Sep 10 2026 at 16:09):if you ask me its probably the safest time, cuz no code changes, any faults with the test, its probably a bad test
Philip Durbin 🚀 (Sep 10 2026 at 16:09):no strong opinion
Munoz Logan Ryan (Sep 10 2026 at 21:31):I’m looking for a team that values collaboration, technical excellence, open communication, and continuous learning.
Ash Manda (Sep 16 2026 at 16:56):Just wanted to give a minor heads up. So far I've noted two tests (really three) tests that seem to be consistently failing on develop
Philip Durbin 🚀 (Sep 16 2026 at 17:14):Thanks for the heads up. I linked to it in Slack.
Philip Durbin 🚀 (Sep 16 2026 at 17:14):Do we have a video of not being able to create a dataset with custom terms?
Ash Manda (Sep 16 2026 at 17:26):Yes, we do!
Ash Manda (Sep 16 2026 at 17:26):b63ec6f9d32934703b26d1b23820addb886306dc.webm
Philip Durbin 🚀 (Sep 16 2026 at 17:29):Awesome. Thanks. Can you replicate it on https://beta.dataverse.org ?
Ash Manda (Sep 16 2026 at 17:30):I wouldn't be able to, cuz I'm not an admin
Ash Manda (Sep 16 2026 at 17:31):making a license default requires admin
Philip Durbin 🚀 (Sep 16 2026 at 17:31):Ok. But should we open an issue? Is it a legit bug?
Ash Manda (Sep 16 2026 at 17:31):this is a known bug iirc
Philip Durbin 🚀 (Sep 16 2026 at 17:33):Do we know its issue number? I'm looking.
Ash Manda (Sep 16 2026 at 17:33):me too, lemme find it
Ash Manda (Sep 16 2026 at 17:34):https://github.com/IQSS/dataverse/issues/12590
Philip Durbin 🚀 (Sep 16 2026 at 17:34):Yes! Thank you! Good find.
Philip Durbin 🚀 (Sep 16 2026 at 20:39):I'm running tests locally for the first time and getting a lot of failures. Does this look ok for my .env?
BASE_URL=http://localhost:8080
DV_USERNAME=dataverseAdmin
DV_PASSWORD=admin1
DV_FULL_NAME=Dataverse Admin
LOGIN_ADAPTER=builtin
Here's the report:
Philip Durbin 🚀 (Sep 16 2026 at 20:40):I'm on #12718 at the moment but I'm happy to switch to the develop branch.
Ash Manda (Sep 16 2026 at 20:40):could you try adding SKIP_PREFLIGHT=true
Ash Manda (Sep 16 2026 at 20:41):if that doesn't fix most of the failures, I'm happy to get on a Zoom call
Philip Durbin 🚀 (Sep 16 2026 at 20:41):Trying it! Thanks!
Philip Durbin 🚀 (Sep 16 2026 at 20:57):5 instead of 6 failures. Progress! :sweat_smile:
Ash Manda (Sep 16 2026 at 20:57):Have to investigate the videos and traces then. That's not normal
Philip Durbin 🚀 (Sep 16 2026 at 20:57):
Philip Durbin 🚀 (Sep 16 2026 at 20:58):mañana :smile:
Ash Manda (Sep 16 2026 at 20:58):its 5pm, wanna do it tmr?
Ash Manda (Sep 16 2026 at 20:58):Philip Durbin 🚀 said:
mañana :smile:
nah I wouldn't be worried
Ash Manda (Sep 16 2026 at 20:58):tbh so far y'all have been assuming that if a test fails, code is wrong. But my incompetence is not entirely out of the question so.
Philip Durbin 🚀 (Sep 17 2026 at 13:24):@Ash Manda sure, I'm happy to do a call whenever. Do you want me to switch to develop first?
Ash Manda (Sep 17 2026 at 13:36):sure, a dry run on develop will give us a baseline!
Ash Manda (Sep 17 2026 at 13:38):Are you available at 10am?
Philip Durbin 🚀 (Sep 17 2026 at 13:40):Sure!
Philip Durbin 🚀 (Sep 17 2026 at 13:40):Should I delete my docker-dev-volumes before running the JSF tests? Or does it not matter?
Ash Manda (Sep 17 2026 at 13:40):it should not matter for the frontend tests at least
Philip Durbin 🚀 (Sep 17 2026 at 13:41):Ok, I deleted it anyway, just in case. :smile:
Oliver Bertuch (Sep 17 2026 at 13:42):@Ash Manda are you rebuilding the base image for JSF tests on feature branches, too? Otherwise the updated Payara would not be included inside the test's app image.
Ash Manda (Sep 17 2026 at 13:43):Yes we've run into that exact issue before. It tries to pull the base image from Docker. If Payara has been upgraded, it wouldn't be able to pull, and it defaults to building the base image
Oliver Bertuch (Sep 17 2026 at 13:46):Smart choice! :smile:
Ash Manda (Sep 17 2026 at 13:47):You can thank Phil for the idea!
Philip Durbin 🚀 (Sep 17 2026 at 13:48):It was born out of a real world failure, during the last Payara upgrade. :smile:
Philip Durbin 🚀 (Sep 17 2026 at 13:55):Switching to develop lowered the failure count from 5 to 4:
Ash Manda (Sep 17 2026 at 14:29):fyi, these failures occurred due to unintended behavior in the suite where it defaults the root dataverse to UNC's unless you explicitly tell it not to: https://github.com/gdcc/dataverse-jsf-tests/issues/7
Philip Durbin 🚀 (Sep 17 2026 at 14:34):Right. I'll probably switch to this:
ROOT_DATAVERSE="/dataverse/root"
To match what we use in CI: https://github.com/IQSS/dataverse/blob/0b292c798581d2f512818d7a9042ba27017f9caf/.github/workflows/dataverse_jsf_tests.yml#L230
Philip Durbin 🚀 (Sep 17 2026 at 14:35):"43 passed (23.2m)" :tada:
Philip Durbin 🚀 (Sep 17 2026 at 14:43):Philip Durbin 🚀 said:
It's a gift! Free code! :tada:
Want me to make an issue about moving the code over?
I checked with @Ash Manda about this and a decision should be made by UNC on Oct 2 about moving the JSF tests to the main/upstream repo.
Ash Manda (Sep 17 2026 at 18:18):Hello. Good afternoon. A merge commit was made into gdcc/dataverse-jsf-tests to introduce the new expanded test suite going from about ~40 tests across three browsers to about ~80 tests across three browsers.
Please let me know if you start noticing any new failures in the mainstream repo
The test development is now going to enter a freeze. Future updates anticipated:
Philip Durbin 🚀 (Sep 17 2026 at 18:19):I passed this along in Slack.
Ash Manda (Sep 17 2026 at 19:50):Just a gentle note for the team. The JSF tests are still in a very very early age of development, they're running in shadow mode right now, basically I'm trying to collect as much data as I can possible to stabilize them, please do not consider the failure of these tests a blocker for merging, for now at this stage, they're an early guide.
Ash Manda (Oct 03 2026 at 18:20):I opened a new PR to address some concerns raised by Oliver and some concerns that I myself had: https://github.com/IQSS/dataverse/pull/12783
For a quick summary: This PR combines the two container test workflows into one, "Containerized Tests for Dataverse". It builds the containers once and runs the API tests and the three JSF browsers in parallel, so JSF results arrive 46% sooner (about 27 minutes instead of 50) with the same test results.
The build each run tested against is now kept for 7 days, so anyone can start it locally to reproduce a failure; the old workflow files are deleted, and the docs and release note are updated.
I welcome all reviews and comments!
Philip Durbin 🚀 (Oct 05 2026 at 13:45):Added to "ready for triage"! Thanks for the PR, @Ash Manda and @Oliver Bertuch.
A little while ago I floated the idea of spinning up the containers once and running both sets of test so I'm glad to see you made it happen! :tada:
Have any changes been made to how we calculate code coverage? Perhaps I should start a new topic about code coverage.
Ash Manda (Oct 05 2026 at 13:45):Not yet, but I'm laying the groundwork for collecting all the stats we need
Ash Manda (Oct 05 2026 at 14:00):Just to clarify, both sets of tests don't run on the same dataverse instance. The containers are built exactly once. Four GitHub Runners download the built artifact and spin up four different instances of Dataverse and run their tests independently on those instances.
Philip Durbin 🚀 (Oct 05 2026 at 14:07):Oh, four instances, ok.
Philip Durbin 🚀 (Oct 05 2026 at 14:07):I kicked off a separate topic about code coverage: #dev > code coverage
Last updated: Nov 02 2026 at 07:01 UTC