Ash Manda (Feb 24 2026 at 20:23): Ash Manda (Feb 24 2026 at 20:23):Hello, Dataverse Team!
My name is Ash Manda. I am an MSCS student and a GRA working in the Infrastructure team of the Research Data Management Core here at UNC-CH. I'm currently working under @Don Sizemore on a variety of tasks involving Dataverse, primarily developing a containerized Playwright testing suite that plugs directly into Maven.
I wanted to reach out and introduce myself since I'll be looking into the current Selenium and Cypress tests in the main Dataverse repo. I will most definitely be asking for some direction on things like handling 2FA as I dig in! :smile:
@Philip Durbin 🚀 @Oliver Bertuch
Philip Durbin 🚀 (Feb 24 2026 at 20:24):@Ash Manda hi!
Philip Durbin 🚀 (Feb 24 2026 at 20:25):Playwright? I thought we had standardized on Cypress. :thinking:
Ash Manda (Feb 24 2026 at 20:37):Hi Philip. Thank you so much for replying. In our use case, we decided to go with Playwright in Java for a multitude of reasons that include some of the following:
Philip Durbin 🚀 (Feb 24 2026 at 21:08):But the testing code should (eventually) be merged into the main repo, right? Should the maintainers of that repo have a say in which tool to use?
Don Sizemore (Feb 25 2026 at 13:44):Philip Durbin 🚀 said:
But the testing code should (eventually) be merged into the main repo, right? Should the maintainers of that repo have a say in which tool to use?
absolutely, and yes we'd like to eventually merge what Ash produces. I just searched the IQSS org and Dataverse guides for cyprus but didn't find much on automated testing there. Is there a document outlining current automated testing plans with Cyprus, would y'all be willing to share that?
Oliver Bertuch (Feb 25 2026 at 14:02):Isn't it "cypress" instead of "cyprus"?
Oliver Bertuch (Feb 25 2026 at 14:02):At least that's what I see in https://github.com/IQSS/dataverse-frontend/blob/develop/cypress.config.ts
Don Sizemore (Feb 25 2026 at 14:28):well, now that I can spell, I have more to go on :rofl: @Ash Manda let's take a look at https://github.com/IQSS/ondemand-loop/blob/6a7df45259fd97d58c9a87b445a8db8d2f99424d/docs/guide/content/development_guide/github_actions.md?plain=1#L10 ?
Philip Durbin 🚀 (Feb 25 2026 at 14:59):Sorry, sorry, I was in a meeting.
Philip Durbin 🚀 (Feb 25 2026 at 15:00):Should we fire up zoom at talk a bit about automated testing?
Philip Durbin 🚀 (Feb 25 2026 at 15:00):I just want to make sure we're on the same page.
Philip Durbin 🚀 (Feb 25 2026 at 15:01):I'm very excited about more automated testing! And fresh ideas!
Don Sizemore (Feb 25 2026 at 15:03):@Philip Durbin 🚀 Ash isn't here just yet today but yes I'd love to put the two of you in touch. We have a specific use case down here (21CFRPart11 validation) but by the time we work that out, we've done the bulk of the work for the greater repo/community.
Philip Durbin 🚀 (Feb 25 2026 at 15:04):Perfect! Thanks!
Philip Durbin 🚀 (Feb 25 2026 at 15:07):Don Sizemore said:
Is there a document outlining current automated testing plans with Cypress, would y'all be willing to share that?
Hmm, I think the closest might be the Dataverse - SPA Development Strategy doc, first shared here. There's a section on testing there that mentions Cypress.
@Ellen K might know more about why we chose Cypress over Playwright for the frontend. I remember talking about it but I can't find any notes.
Of course, for the backend we already use REST Assured for API testing. This is described here: https://guides.dataverse.org/en/6.9/developers/testing.html#writing-api-tests-with-rest-assured
Philip Durbin 🚀 (Feb 25 2026 at 15:08):One of the things I'd like to understand is what we'd use Playwright for in the backend repo. Would we be testing the JSF code that we plan to throw away once we've switched to the SPA? :trash_can: :thinking:
Don Sizemore (Feb 25 2026 at 17:58):@Philip Durbin 🚀 good afternoon! we're free until 3pm if you want me to scare up Ash for a Zoom?
Philip Durbin 🚀 (Feb 25 2026 at 18:20):Hmm, I have a meeting in 10 minutes but we could talk quick if you want. Or maybe at 2pm?
Don Sizemore (Feb 25 2026 at 18:20):2pm would be perfect!
Philip Durbin 🚀 (Feb 25 2026 at 18:21):Great, hopefully my 2FA is fixed by then :grimacing:
Philip Durbin 🚀 (Feb 25 2026 at 19:50):Thanks for chatting! See you at the frontend meeting tomorrow!
Philip Durbin 🚀 (Feb 26 2026 at 12:32):I just added this topic to the agenda: https://docs.google.com/document/d/1ape8WJZjkjyypnHB88Ieodv7BSMJGVyZYlYwAdrZUM8/edit?usp=sharing (please feel free to request access).
Philip Durbin 🚀 (Feb 26 2026 at 16:59):@Don Sizemore @Ash Manda thanks for joining! This is what I wrote in the notes:
Philip Durbin 🚀 (Feb 26 2026 at 17:01):I'll just add that I'd kindly request you to delete the old Cypress tests added in https://github.com/IQSS/dataverse/pull/5922 as part of this effort. Sadly, we never got a chance to make proper use of them.
Philip Durbin 🚀 (Feb 26 2026 at 17:02):You might find inspiration from the Cypress tests at https://github.com/IQSS/dataverse-frontend under the "tests" directory. And DEVELOPER_GUIDE.md talks about them.
Philip Durbin 🚀 (Jun 04 2026 at 13:34):@Ash Manda @Don Sizemore I seem to remember that you pushed a repo to GitHub with Cypress tests for JSF but I can't find it. Do you know where it is? Thanks!
Ash Manda (Jun 04 2026 at 13:45):Hi Phil. Cypress tests? The tests that I pushed, I wrote in Playwright because Cypress couldn't get past Shibboleth. Maybe there's something else that you're talking about?
Ash Manda (Jun 04 2026 at 13:46):Since you said that the repo was standardized in Cypress, we didn't want to force push that onto y'all so we went ahead and made it private
Philip Durbin 🚀 (Jun 04 2026 at 13:48):Oh, so what about the dream of having automated tests for JSF merged upstream?
Ash Manda (Jun 04 2026 at 13:50):https://youtu.be/jRPLoMMPwxI?si=oeIQVGj939T4KaUg
Philip Durbin 🚀 (Jun 04 2026 at 13:50):Philip Durbin 🚀 said:
Don Sizemore Ash Manda thanks for joining! This is what I wrote in the notes:
- Decision: Write Cypress tests against JSF in the backend repo. Eventually, these tests will be moved to the frontend repo, once the SPA is ready.
- What about logging in with 2FA in GitHub Actions? This might be a problem.
What?!?! :crazy: I thought we had a deal! ^^ :crazy:
Philip Durbin 🚀 (Jun 04 2026 at 13:51):Should I ask you for a PR to put the Playwright tests in the upstream repo?
Philip Durbin 🚀 (Jun 04 2026 at 13:51):I care much more about automated tests that what tool is used.
Philip Durbin 🚀 (Jun 04 2026 at 13:52):Especially as release time approaches! #community > Release 6.11 Timeline
Ash Manda (Jun 04 2026 at 13:52):Well they're very specifically built for Shibboleth and our secure data store. If Don wants me to retool them, I can try
Philip Durbin 🚀 (Jun 04 2026 at 13:53):I put in on the agenda for next week's dev meeting (tech hours) at least!
Philip Durbin 🚀 (Jun 04 2026 at 13:53):Let's see if others want these tests too. (I assume they do!)
Ash Manda (Jun 04 2026 at 14:08):Some of the tests I designed are capable of punching through Anubis and most WAF setups, personally I would prefer they remain private, but let me know.
Philip Durbin 🚀 (Jun 04 2026 at 15:34):Sure, but maybe we could get a few tests out of you! :smile:
Philip Durbin 🚀 (Jun 22 2026 at 15:35):Philip Durbin 🚀 said:
I put in on the agenda for next week's dev meeting (tech hours) at least!
@Ash Manda sorry we didn't get to this topic during the meeting but I have it on the agenda for tomorrow. Any chance you can join?
Ash Manda (Jun 22 2026 at 16:35):Yep, Don asked me to help in whatever way possible, I'd be happy to join!
Ash Manda (Jul 01 2026 at 16:32):Hi. After a lot of thinking, and scrubbing our testing repo, I think I might have an idea on how IQSS can run these frontend tests without ever touching the test source code itself.
Ash Manda (Jul 01 2026 at 16:33):It will prevent y'all from having a submodule/subtree headache
Ash Manda (Jul 01 2026 at 16:37):To allow upstream collaboration and open-source testing without exposing proprietary infrastructure, specifically our Web Application Firewall (WAF) bypass mechanisms, I propose decoupling our test cases from our execution scaffolding.
We will transition from a single private repository to a dual-repository architecture. This guarantees that test logic is public and trackable by upstream, while our specialized testing infrastructure remains strictly private.
Content: Contains only the core test files (/tests) and a standard, vanilla Playwright configuration.
Security Posture: Public. Absolutely no WAF bypass, custom proxies, or such dependencies will exist here.
Purpose: Serves as the public source of truth for upstream contributors to view, run, and update tests using standard web automation.
Content: Contains our proprietary execution scaffolding, WAF bypass logic, and other integrations.
Mechanism: We will pull only the /tests folder from the public repository into this private environment (via Git submodule mapping or sparse-checkout automation).
Purpose: Allows our internal team to run the exact same public test cases, but routed through our specialized, private evasion infrastructure.
To automate this for the upstream repository at Harvard, we will implement a lightweight GitHub Actions workflow for the main repo:
[Upstream Trigger] ──> [Clone Public Test Repo] ──> [Spin up App via Docker] ──> [Run Vanilla Playwright] ──> [Report Results]
Trigger: A Pull Request or commit occurs in the upstream repository.
Environment Setup: The GitHub Action clones the Public Test & Vanilla Scaffolding Repo.
Application Deployment: The workflow builds and runs the upstream application inside a localized Docker container.
Execution: The standard Playwright suite runs against the local Docker container.
Reporting: Artifacts (test logs/results) are saved, and the pass/fail status is reported directly back to the upstream PR.
Zero Intellectual Property Leakage: Our specialized browser tooling and WAF bypass logic are physically isolated in a private environment.
Native Upstream Integration: Upstream maintains a completely standard, dependency-free testing pipeline via GitHub Actions.
Bidirectional Collaboration: If upstream fixes or adds a test in the public repo, our private repo pulls the update instantly without breaking our internal scaffolding.
Ash Manda (Jul 01 2026 at 16:40):The only catch is that it'll take me a while, I got a lot on my plate
Philip Durbin 🚀 (Jul 01 2026 at 17:41):This plan reminds me a lot of https://github.com/gdcc/api-test-runner where we had a repo on the side that we could trigger at any time to run tests. Sounds fine to me!
Ash Manda (Jul 02 2026 at 20:42):We have a working GitHub Action here: https://github.com/uncch-rdmc/dataverse/actions/runs/28618020588/job/84870297997
The test videos, screenshots, traces, reports are all available for download
The repository being pulled for testing is: https://github.com/uncch-rdmc/kunai-runner
Some of the tests (especially one specific test) are a bit flakier than expected because they weren't really originally designed to be tested on the upstream develop branch with a generic configuration on a GitHub Actions runner, but I'm working on them to make them better. Feedback welcome before a PR is made! You can expect an issue and PR to be opened next week.
Philip Durbin 🚀 (Jul 02 2026 at 20:50):These videos are great!
3251d26c2a0718db1b2030139f5670723754389c.webm
Philip Durbin 🚀 (Jul 02 2026 at 20:57):And I love the dashboard:
Ash Manda (Jul 06 2026 at 14:46):We got a pass folks: https://github.com/uncch-rdmc/dataverse/actions/runs/28618020588/job/85392382694
Philip Durbin 🚀 (Jul 06 2026 at 14:46):Fantastic!
Philip Durbin 🚀 (Jul 06 2026 at 14:47):@Ash Manda do you have time to talk about this at tech hours tomorrow?
Ash Manda (Jul 06 2026 at 14:48):Sure!
Philip Durbin 🚀 (Jul 06 2026 at 14:48):Great! Added to the agenda! https://docs.google.com/document/d/1xVD_Xi5OYlJnU1scEDQv5jjEIovu1GZDuilxwpgorpo/edit?usp=sharing
Philip Durbin 🚀 (Jul 06 2026 at 14:48):Please feel free to add whatever links or whatever.
Philip Durbin 🚀 (Jul 06 2026 at 14:49):And please request edit access, as needed.
Ash Manda (Jul 07 2026 at 19:27):Issue opened here: https://github.com/IQSS/dataverse/issues/12525
PR opened here: https://github.com/IQSS/dataverse/pull/12526
Always happy for feedback, suggestions, and to address concerns. Thank you!
Philip Durbin 🚀 (Jul 07 2026 at 19:27):So fast! Thanks, @Ash Manda!! 
Ash Manda (Jul 07 2026 at 19:39):Philip Durbin 🚀 said:
I'll just add that I'd kindly request you to delete the old Cypress tests added in https://github.com/IQSS/dataverse/pull/5922 as part of this effort. Sadly, we never got a chance to make proper use of them.
Removing these tests may also fall squarely under the scope and purview of this PR since they're both meant to test JSF.
Philip Durbin 🚀 (Jul 07 2026 at 20:45):Yes! Please feel free!
Philip Durbin 🚀 (Jul 10 2026 at 19:54):@Ash Manda do you want me to remove them? In #12526 I mean.
Ash Manda (Jul 10 2026 at 20:05):Of course! I think you'll be the best person to do it. I held off from doing it because I remember in the last tech hours some team members said that they still regularly use those Cypress tests so I held off
Philip Durbin 🚀 (Jul 10 2026 at 20:06):Nobody uses those particular tests in the backend repo.
The frontend repo uses Cypress tests, yes.
Ok, I'll see what I can do.
Don Sizemore (Jul 10 2026 at 20:08):I dropped the ball here on double-checking and reporting back to Ash. We've been banging our heads on Globus in Glacier. We're also going to move the Kunai repo into the GDCC GitHub org.
Philip Durbin 🚀 (Jul 10 2026 at 20:08):Great!
Philip Durbin 🚀 (Jul 10 2026 at 20:08):Not the banging heads part. Ouch!
Ash Manda (Jul 10 2026 at 20:09):If there are any new naming ideas for the repository, this is a good time!
Philip Durbin 🚀 (Jul 10 2026 at 20:09):dataverse-jsf-tests
Don Sizemore (Jul 13 2026 at 19:33):Philip Durbin 🚀 said:
dataverse-jsf-tests
done! https://github.com/gdcc/dataverse-jsf-tests
Philip Durbin 🚀 (Jul 13 2026 at 19:36):@Don Sizemore thanks! Now we need to update the PR, right? I left a reminder: https://github.com/IQSS/dataverse/pull/12526#pullrequestreview-4688100442
Ash Manda (Jul 13 2026 at 19:45):I went ahead and updated both the PR and the original issue
Philip Durbin 🚀 (Jul 14 2026 at 20:28):@Ash Manda thanks! Next up, can you please help with this error I'm seeing? https://github.com/IQSS/dataverse/pull/12526/changes#r3582393242
Philip Durbin 🚀 (Jul 14 2026 at 20:29):I'm curious where I can find this video:
attachment #2: video (video/webm)
test-results/13-dataset-actions-21-CFR--de5bc-reate-edit-replace-publish--suite-webkit/video.webm
Philip Durbin 🚀 (Jul 14 2026 at 20:32):Ah, probably "playwright-report" at https://github.com/IQSS/dataverse/actions/runs/29360743780
Philip Durbin 🚀 (Jul 14 2026 at 20:35):Ah, "Test timeout of 180000ms exceeded":
Philip Durbin 🚀 (Jul 14 2026 at 20:36):From the video, it seems to hang at this step:
Ash Manda (Jul 14 2026 at 20:39):This is some flakiness I noticed specifically with GitHub Actions
Ash Manda (Jul 14 2026 at 20:39):The default Ubuntu runner is just barely powerful enough for Dataverse, so a lot of the actions lag and may or may not finish as intended, which I've never noticed on a standard installation
Ash Manda (Jul 14 2026 at 20:40):I pushed a fix right now, hoping to see if that fixes it
Philip Durbin 🚀 (Jul 14 2026 at 20:40):Oh dear. What do you mean by "standard installation"?
Ash Manda (Jul 14 2026 at 20:41):Like an EC2 server that has Dataverse installed the classic way, not using Docker.
Ash Manda (Jul 14 2026 at 20:41):The Docker one has a lot of weirdness I noticed. For example. If I add a user through the Dataverse API, make them superuser, and then try to log in with that user, it just wouldn't log in
Ash Manda (Jul 14 2026 at 20:44):The specific failure that I'm talking about, you can view it in action here: https://github.com/uncch-rdmc/dataverse/actions/runs/28617341949
I was able to replicate it using the demo docker-compose.yml file provided by Harvard. So instead I had to rewrite the Actions to use dataverseAdmin instead
Philip Durbin 🚀 (Jul 14 2026 at 20:46):If we're already right on the edge of Dataverse being too fat to run in the default Ubuntu runner, I'm nervous about adding these JSF tests as a GitHub Action. :grimacing: :piglet:
Ash Manda (Jul 14 2026 at 20:47):It's a very valid concern, one that even I had. But, at the end of the day like you yourself said Phil, its better to have some running tests than none
Oliver Bertuch (Jul 14 2026 at 20:49):Guys my phone is blowing up here. So let me ask a few things...
Oliver Bertuch (Jul 14 2026 at 20:49):Ash, you did assign more memory to the DV container?
Oliver Bertuch (Jul 14 2026 at 20:50):Have you tried playing with the heap ratio setting?
Ash Manda (Jul 14 2026 at 20:50):Not yet.
Oliver Bertuch (Jul 14 2026 at 20:51):AFAIK the Ubuntu runners provide 7GB RAM and a few burst CPU. It sounds like this should be enough, given we can run most of our API tests with a 2.5? 3? GB minimum
Ash Manda (Jul 14 2026 at 20:52):It's the JSF frontend clicking and loading that I've observed lagging
Ash Manda (Jul 14 2026 at 20:52):Ash Manda said:
The specific failure that I'm talking about, you can view it in action here: https://github.com/uncch-rdmc/dataverse/actions/runs/28617341949
I was able to replicate it using the demo docker-compose.yml file provided by Harvard. So instead I had to rewrite the Actions to use dataverseAdmin instead
That and this failure.
Oliver Bertuch (Jul 14 2026 at 20:53):The demo compose file only uses memory sparingly.
Oliver Bertuch (Jul 14 2026 at 20:53):Memory is at a premium these days and your tests are probably no joke.
Ash Manda (Jul 14 2026 at 20:54):They haven't completely broken down yet, they run fine 99% of the time, so I think we're good as long as I make the waiting more robust. And that I can only do with more runs
Oliver Bertuch (Jul 14 2026 at 20:57):Might be interesting to use one of the two telemetry actions to measure resource usage of the runner VM.
Oliver Bertuch (Jul 14 2026 at 20:58):https://github.com/marketplace?query=telemetry&type=actions
Oliver Bertuch (Jul 14 2026 at 20:58):Measure if this is a resource limitation the runner imposes or a setup issue.
Ash Manda (Jul 14 2026 at 20:59):I might add in an OTel action and hook it up to Grafana, it feels like an important thing to have
Ash Manda (Jul 14 2026 at 21:00):Does IQSS have an org Grafana account?
Oliver Bertuch (Jul 14 2026 at 21:01):Not as far as I know but they have a free tier for their cloud offering.
Ash Manda (Jul 14 2026 at 21:01):Okay let me poke around on that. Thank you!
Philip Durbin 🚀 (Jul 14 2026 at 21:01):Ash Manda said:
Does IQSS have an org Grafana account?
Not that I know of.
Philip Durbin 🚀 (Jul 14 2026 at 21:02):Thanks, all!
Oliver Bertuch (Jul 14 2026 at 21:02):I'm happy to create a gdcc thing for this. Let me know if that would help.
Ash Manda (Jul 14 2026 at 21:03):It absolutely would! All the GitHub Actions in upstream can receive monitoring then. (As long as Grafana doesn't get greedy enough)
Ash Manda (Jul 14 2026 at 21:40):Philip Durbin 🚀 said:
Ash Manda thanks! Next up, can you please help with this error I'm seeing? https://github.com/IQSS/dataverse/pull/12526/changes#r3582393242
I ran a test with the updated fix here: https://github.com/uncch-rdmc/dataverse/actions/runs/29367972485/job/87204182793
It seems to be working fine now. Please feel free to rerun the workflow on upstream from your side. It won't auto-trigger, because the actual testing repo is decoupled. Thank you!
Philip Durbin 🚀 (Jul 15 2026 at 01:07):Ok, so this fix: https://github.com/gdcc/dataverse-jsf-tests/commit/84fa93d29d5a0fa8195a47c95d1363e94ab7448b
Ash Manda (Jul 15 2026 at 01:24):Yep. That's the one! Instead of hoping it is visible just in time, I just forced it to make sure everything is visible step by step.
Oliver Bertuch (Jul 15 2026 at 08:25):@Ash Manda are you planning on contributing the Action Workflow upstream? If so, would you mind renaming it and drop the "container" from the name? The "container" actions are centered around building and distributing the container images. It may confuse people as you're mostly a user of them container images.
Ash Manda (Jul 15 2026 at 10:40):Yes, there's a PR open to contribute it upstream. I can change the name yeah, I just needed some way to indicate quickly that these tests are running in a containerized Dataverse. Do you have any suggestions for a name? Maybe JSF-Frontend-Tests-Docker?
Philip Durbin 🚀 (Jul 15 2026 at 12:43):@Oliver Bertuch are you asking about this PR? #12526
Ash Manda (Jul 15 2026 at 18:30):Renamed to Dataverse JSF Tests/Dataverse Frontend JSF tests to reduce ambiguity regarding containers.
Philip Durbin 🚀 (Jul 15 2026 at 18:33):Does that mean I can merge it? :smile:
Philip Durbin 🚀 (Jul 15 2026 at 20:18):Merged! #12526
Ash Manda (Jul 15 2026 at 21:33):Awesome, thank you! @Don Sizemore will be so happy, it was a major goal for him.
Last updated: Aug 18 2026 at 08:27 UTC