Frank Smutniak (May 18 2026 at 16:58): Frank Smutniak (May 18 2026 at 16:58):Our production dataverse runs 6.10.1 on an 8 virtual CPU server with 32GB memory.
We want to make sure our settings make full use of the resources.
We build the server using the Harvard ansible scripts but expect there are
some server specific adjustments we need to make beyond initial setup.
Our current settings look like the following:
domain1/config/domain.xml
<jvm-options>-Xmx16384m</jvm-options>
For domain1/config/domain.xml what are suggested min-thread-pool-size and max-thread-pool-size for the <thread-pool>
associated with http-listener-1 (http-thread-pool)
We currently do not have these set but are considering:
<jvm-options>-XX:MaxMetaspaceSize=4096m</jvm-options>
<jvm-options>-XX:MetaspaceSize=2048m</jvm-options>
<jvm-options>-Xms12288m</jvm-options>
In solr/bin/solr.in.sh we currently have no MEM settings. Should we consider?
SOLR_JAVA_MEM="-Xms4g -Xmx4g"
For apache, is a connection pool set of 300 optimal?
For postgres, are there settings you've found to be optimal for working with dataverse?
Thank you for any guidance.
Don Sizemore (May 22 2026 at 15:42):Hello,
I wouldn't try to run Dataverse in a JVM heap smaller than 16g as you've specified. Dataverse usage can be bursty; the greatest memory spike will happen during tabular ingest.
I've never played with the min/max thread pool sizes for the http listeners, but I remember reading that MaxMetaspaceSize and MetaspaceSize should be set to the same value to avoid performance problems (this may now be historic).
UNC Dataverse currently runs in a Solr JVM heap setting of 1g; 4g should be plenty.
The connection pool size for Apache depends on your usage; I'd imagine 300 should be plenty as long as the LLM scrapers stay away.
I've never had to tune Postgres; our production RDS instance has 2 CPU and 8GB of RAM; the biggest single effect I've seen on performance depends on administrators calling ANALYZE; (or not) following a database import.
I'd love to hear what other community members have to say here.
Don
Thomas van Erven (May 23 2026 at 07:29):Most "tuning" I needed to do on Postgres was cleanup a rather large ACTIONLOGRECORD table (years of buildup) and a lot of notifications. Runs fine under default settings, Postgres being a saint and all.
I run demo/test dataverses on smaller VM's, but despite best effort, I find Payara has serious difficulty getting out of bed for less than 2 GB heap minimal and the sweet spot is somewhere around 3-4 GB.
Workloads are on the spiky side of ops as Don indicates, tabular ingest can be an evil sod occasionally as users will happily add large tabular data files. Treat any post-processing (workflows) you do as those Mario enemies with spikes on all sides - expect to see trouble from them at least once even if they look innocent.
Thomas van Erven (May 23 2026 at 07:32):One thing I'd like to do at some point is mess with a pgvector database, for that matter. Having a db that can natively store embeddings in a relatively efficient way to act as a search index (rather than Payara) might be trading disk space for memory but that's a decent deal in this case. I had some good luck with FFF-driven vector space being a relatively cheap win with searching.
Frank Smutniak (May 26 2026 at 15:22):Thanks Don and Thomas,
The bots are a big drain on resources in bursts but with our recent changes it is having less downtime or slow service. Thank you for your suggestions and confirmation.
Frank
Frank Smutniak (May 26 2026 at 15:23):Removing ACTIONLOGRECORD was done as we recently ported from the old amazon linux to AL 23. It saved a lot of space!
Thomas van Erven (Jun 03 2026 at 07:15):Having spoken about it with some folks at this point; did you consider Anubis to block off bots? It's a bit of a mixed blessing since researchers will also use AI to go through your Dataverse (so it's "legit" usage but simply at a scale we're not used to).
Thomas van Erven (Jun 03 2026 at 07:20):I reckon we're going to be stuck with that for a while still, and I'm hearing people consider it legit usage by researchers (thus scaling upwards to meet demand) and some people going "nope" and deploying Anubis to save costs.
Has anyone implemented some sort of clever MCP server discovery+redirect to point bots to the right entrance for these kind of tasks?
Philip Durbin 🚀 (Jun 03 2026 at 11:00):You're thinking that an MCP-based solution would be easier on the servers? Rather that the servers being crawled, that is?
Oliver Bertuch (Jun 03 2026 at 12:49):Is MCP really the way to go here? Wouldn't an agents.md be a more fitting approach? https://agents.md/ (there's also SKILL.md)
Thomas van Erven (Jun 04 2026 at 13:37):I think web UI requests are likely more expensive, sure. I also don't know if MCP is the way to go but I like to point bots to the bot entrance if it's cheaper and more fitting to use
Thomas van Erven (Jun 04 2026 at 13:38):I'm just spit balling for now but the discussion I've had so far chiefly goes around 'at what cost can we facilitate AI users' so cost as a metric seems the way to look at that
Philip Durbin 🚀 (Jun 04 2026 at 13:40):Pull request #12025 by @Juan is about telling bots where to find APIs and standards used, using .well-known/api-catalog from https://datatracker.ietf.org/doc/rfc9727/
Oliver Bertuch (Jun 08 2026 at 19:42):https://contentsignals.org/ seems very much related!
Oliver Bertuch (Jun 08 2026 at 19:43):https://modelcontextprotocol.io/community/server-card/charter#deliverables-%26-success-metrics says they don't have a final spec yet. (Draft at https://github.com/modelcontextprotocol/modelcontextprotocol/pull/2127, .well-known/mcp.json)
Oliver Bertuch (Jun 08 2026 at 19:46):See also https://blog.cloudflare.com/agent-readiness/
Philip Durbin 🚀 (Jun 08 2026 at 19:50):cool
Oliver Bertuch (Jun 08 2026 at 19:52):I'm not sure serving .well-known/api-catalog is a job for Dataverse directly. Very often you'll expose other .well-knowns, so maybe it should be an internal rewrite from the reverse proxy to an "normal" API endpoint /api/info/api-catalog or similar.
Oliver Bertuch (Jun 08 2026 at 19:52):The proposed way to integrate it into a dataset page is quite costly.
Thomas van Erven (Jun 10 2026 at 09:00):Would it be reasonable to handle it like sitemap generation to avoid costly hits? (started by user with admin access on the API, generates files which can be served by webserver (not payara), and give a few examples on how the reverse proxy should behave)
Secondary reason why that might be useful is because I don't see these types of requests decreasing in the future, so offloading to battle-tested webserver to serve files seems the right call.
Philip Durbin 🚀 (Jun 10 2026 at 10:49):I don't see them deceasing either.
Notification Bot (Jul 07 2026 at 17:50):This topic was moved here from #troubleshooting > Optimal settings for dataverse and its support stack by Philip Durbin 🚀.
Last updated: Aug 18 2026 at 08:27 UTC