Ash Manda (Oct 14 2026 at 14:12): Ash Manda (Oct 14 2026 at 14:12):Hello and good morning. Do we know if this repo is being used at the moment anywhere? https://github.com/gdcc/api-test-runner
gh search code says that its a downstream consumer of integration-tests.txt
Philip Durbin 🚀 (Oct 14 2026 at 14:19):It's been broken for years. Please don't worry about it.
Ash Manda (Oct 14 2026 at 15:16):@Oliver Bertuch I feel like. If we move from .txt to @API. We're re-introducing the problem, but in a different flavor. There's nothing wrong with the file now except for the fact that you have to manually add tests to the file so that they're run. If we start adding tags to these tests, say a dev is pushing an emergency fix in the night and forgets a tag, the issue might be fixed at the moment, but its not gonna help us going forward. I want to see auto-discovery. If we're going to do this, let's do it exactly once in the right way. Ideally we'd want to reverse the default.
Ash Manda (Oct 14 2026 at 15:19):Maybe a new tag saying CI_EXCLUDED?
Ash Manda (Oct 14 2026 at 15:23):An example
public static final String CI_EXCLUDED = "ci-excluded";
@Target(TYPE)
@Retention(RUNTIME)
@Tag(Tags.CI_EXCLUDED)
public @interface ExcludedFromCi {
/** Why this class can't pass in the CI container setup. */
String reason();
/** Issue tracking the fix, if any. */
String issue() default "";
}
@ExcludedFromCi(reason = "needs review.tsv, which is not loaded out of the box")
public class ReviewsIT { ... }
In the pom -Pci-api-tests.:
<it.groups.ci>!testcontainers & !migration & !performance & !ci-excluded</it.groups.ci>
mvn test-compile failsafe:integration-test failsafe:verify -Pci-api-tests \
-Ddataverse.test.baseurl=http://localhost:8080
or even simpler: mvn test -Pci-api-tests and let all the flags for CI exist in the pom
Ash Manda (Oct 14 2026 at 15:29):The pom already has a TODO for it, right above the ct profile: "Add a profile to run API tests (integration tests that end in IT.java...)".
Philip Durbin 🚀 (Oct 14 2026 at 15:52):I'm fine with an "excluded" tag.
Oliver Bertuch (Oct 14 2026 at 21:25):My vision here is somewhat different. I want to be able to run something likemvn verify -Dit.groups=api to simply run all the API tests in one go.
Oliver Bertuch (Oct 14 2026 at 21:26):The problem: we are using "IT" as a suffix for everything.
Oliver Bertuch (Oct 14 2026 at 21:27):Technically, you can do whatever you want. For good practice, unit tests usually end in "Test" and any kind of "integration" test ends in IT. Units tests get run by Maven Surefire, typically bound to the Maven "test" phase.
Oliver Bertuch (Oct 14 2026 at 21:28):Any "IT" by default is to be executed by Maven Failsafe, typically bound to the "verify" phase. Of course, any of the two can ran any test.
Oliver Bertuch (Oct 14 2026 at 21:29):The difference is in how they handle executing the tests: https://www.baeldung.com/maven-surefire-vs-failsafe
Oliver Bertuch (Oct 14 2026 at 21:29):IMHO we should start adopting good practices and not fight the ecosystem, but move with it.
Oliver Bertuch (Oct 14 2026 at 21:30):Using Failsafe's default configuration to auto-discover any "*IT" classes and execute them is getting in the way for us.
Oliver Bertuch (Oct 14 2026 at 21:31):While for self-contained tests in the groups "performance", "integration", etc I made sure they can just be picked up, the same is not true for API tests.
Oliver Bertuch (Oct 14 2026 at 21:31):They require setting up the test instance first, before executing the tests.
Oliver Bertuch (Oct 14 2026 at 21:32):Eventually, I would very much prefer using something inside Maven or JUnit to setup this stack automatically before executing the tests.
Oliver Bertuch (Oct 14 2026 at 21:32):But we are not there (yet).
Oliver Bertuch (Oct 14 2026 at 21:32):So while I agree auto-discovery would be superb, we kind of have to live with our self-inflicted limitations.
Oliver Bertuch (Oct 14 2026 at 21:34):But even if we do auto-discovery, I totally would want to enable people just to run a selective set of tests. Not every test needs to be run all the time on local development, but maybe you want the full set of integration tests now. IMHO it would be nice to still keep these categories alive, but as an addition, not a requirement.
Oliver Bertuch (Oct 14 2026 at 21:36):So how about this: we do auto-discovery, but with a second Failsafe configuration. We invent a new coding standard by calling our API tests just that. Something like DatasetAT or ApiTestDataset or DatasetTestApi. (Not using IT or Test as a suffix would be crucial to avoid the necessity to declare more complex regex for surefire/failsafe).
Oliver Bertuch (Oct 14 2026 at 21:37):That way we get autodiscovery any time someone adds an API test where they don't need to put Tags on things (after all they were meant for more meaningful grouping than just test category).
Oliver Bertuch (Oct 14 2026 at 21:38):We can make the Maven configurations have their own skips, following the established patterns.
Oliver Bertuch (Oct 14 2026 at 21:39):This is all independent of Maven profiles (which we can use to do all sorts of crazy things)
Oliver Bertuch (Oct 14 2026 at 21:46):We could even be so bold to use Maven phases to our advantage! Looking at the default lifecycle, we could bind running any "IT" to the "integration-test" phase and bind any "AT" to the "verify" phase. That would totally be inline with the tooling and good practices. Also, if we ever enable using a Maven/JUnit controlled stack deployment, that totally could be bound to any of the intermediate phases.
Oliver Bertuch (Oct 14 2026 at 21:49):Just to give an example: using the Docker Maven Plugin (or any other plugin that steers containers, like the Docker Compose plugin) we can spin up a compose stack in "post-integration-test" and let any API tests "attack" it in "verify". That's cheap and easy to do, not involving any more complicated testcontainers setup.
Oliver Bertuch (Oct 14 2026 at 21:49):Thanks for reading this far, braindump out.
Ash Manda (Oct 14 2026 at 23:59):Thank you for your detailed response. This makes a lot of sense to me. I agree with the plan, both don't have to be mutually exclusive. Few caveats.
Number one, This means enforcing some kind of standard to developers that they have to manually name classes *AT. If say out of habit they name an API test *IT, and we aren't actively looking for it in the code review phase, it's going to slip by, CI is going to ignore it, and nobody will know.
Number two, My biggest problem with mvn verify has been that it runs the mvn package step. Assuming this is how our testing pipeline is going to be going forward: https://github.com/IQSS/dataverse/pull/12783 mvn verify kills the one thing that I intended to give from this PR which is time savings. In that specific PR, verify when tested rewrote the mounted /target/dataverse container. Until the stack setup itself moves fully into maven, we might have to make some tradeoffs here in terms of best practices and intended lifecycles.
Number three, merge conflicts. Anyone who's actively editing one of these API test classes now is going to be in for a rude shock. We'll have to inform people ahead of time.
So, here's what I might target in scope for three PRs:
Does this sound like a plan?
Philip Durbin 🚀 (Oct 15 2026 at 00:01):I'm totally fine with renaming classes to be in line with best practices. The only reason I started using IT at the end was that in Netbeans I would right-click an API class and click "open test class" and it would create DatasetsIT.java or whatever if it didn't exist. So I went with it. 
Oliver Bertuch (Oct 15 2026 at 07:48):I'm not saying that using "AT" is a best practice. We're inventing our own good practice here out of necessity, allowing our test infrastructure to deal with things more easily.
Here's another alternative to renaming: we could autodiscover based on Java package name. All of our API tests are in one package. It still is a coding standard / convention though, but avoids having a different suffix. My argument for not doing this: I want to create more integration tests for the API based on Jersey Tests. These would be "ITs" and live in the same package as the API tests, so the different suffix would make them distinguishable.
Another crazy thought: what if we create a Maven submodule for the API tests? I've seen that pattern in the wild, too. Not getting rid of another type of codebase convention, but there is no free lunch here.
Philip Durbin 🚀 (Oct 15 2026 at 11:09):Should we write the plan down in a Google doc and send it around for comment?
Ash Manda (Oct 15 2026 at 17:00):We absolutely should, yes!
Oliver Bertuch (Oct 15 2026 at 17:05):Anyone up for a campfire and a fireside chat in a cozy Zoom room to talk about this?
Ash Manda (Oct 15 2026 at 17:06):I am technically on vacation, but let's talk!
Oliver Bertuch (Oct 15 2026 at 17:06):LOL I hear you :sweat_smile: I'm technically done for the day :wink:
Oliver Bertuch (Oct 15 2026 at 17:14):https://fz-juelich-de.zoom-x.de/j/66713629528?pwd=2hbZ4aZmMaisexaLioyBuNEiz8axdn.1
Last updated: Nov 02 2026 at 07:01 UTC