Stream: infra-ops

Topic: Question about infrastructure/server config

Deirdre Kirmis (Jul 07 2026 at 17:31):

Since @Philip Durbin 🚀 was so kind as to create this channel .. I'll ask the first (of likely many) question(s)!

• Will start basic .. sysadmins, what does your infrastructure look like?
• Are you in the cloud, or on-prem?
• What OS do you run Dataverse on?
• Do you have a 1) single server for webapp+solr+ext tools or 2) a server for app, server for microservices, solr, external apps, 3) some other config?
• What are your server specs (ie: cpu, ram, etc)
• Do you run external tools locally on the app or services server, or point to the public repo?

In the spirit of sharing, at ASU we are:

• all in AWS
• running Rocky Linux 8 still
• have a server for web app and a server for SOLR .. but are going to combine soon (w/b m6a.2xlarge - 8CPU/32GRAM)
• we point to repo for some external tools (ie: previewers, data exp) but have some installed locally .. after doing the above will create a "services" server to install/run all locally; we actually have a third instance for Globus transfer app but I will combine that with the other services and tools on the "services" server

Any thoughts on running dataverse+solr on the same server?

I guess that was multiple questions, but hopefully all related.

Philip Durbin 🚀 (Jul 07 2026 at 17:44):

Harvard Dataverse is in the cloud, on AWS.

We run on some variant of Red Hat, either RHEL or Rocky.

We're on multiple servers, described here: https://guides.dataverse.org/en/6.11/installation/prep.html#advanced-installation

Philip Durbin 🚀 (Jul 07 2026 at 17:45):

Here's a pic:

Philip Durbin 🚀 (Jul 07 2026 at 17:46):

Our specs are described here: https://guides.dataverse.org/en/6.11/installation/prep.html#hardware-requirements

Namely:

• two instances for web frontends running Payara fronted by Apache (“m4.4xlarge” with 64 GB RAM and 16 vCPUs)
  • these are sitting behind an AWS ELB load balancer
• one instance for the Solr search engine (“m4.2xlarge” with 32 GB RAM and 8 vCPUs)
• one instance for R (“m4.xlarge” instances with 16 GB RAM and 4 vCPUs)
• The PostgreSQL database is served by Amazon RDS.

Philip Durbin 🚀 (Jul 07 2026 at 17:47):

We point to the public gdcc repo for external tools, like this: https://gdcc.github.io/dataverse-previewers/previewers/v1.5/SpreadsheetPreview.html

Deirdre Kirmis (Jul 07 2026 at 17:55):

Ah yea, I've seen this .. something for us to strive to!
Interesting, so you are running the database on the instance and not in RDS?

And, are those 2 (3?) webserver instances permanent or does the LB utilize autoscaling? And on-demand instances, or reserved? We may need to increase our specs.

And you have SOLR separate .. well maybe we should leave ours as is. Good to know you are using the repo for external tools too.

Thanks for the response! Would love to know what other orgs are doing. I know a few have mentioned that they have webapp+solr on a single server.

Deirdre Kirmis (Jul 07 2026 at 17:56):

Thanks for moving the other issues into this channel .. some I had not seen and a lot of good answers.

Philip Durbin 🚀 (Jul 07 2026 at 17:56):

Deirdre Kirmis said:

Interesting, so you are running the database on the instance and not in RDS?

Oh! Excellent point! That diagram is out of date. We use RDS.

Philip Durbin 🚀 (Jul 07 2026 at 17:57):

I just edited the message above to include this line from the guides:

"The PostgreSQL database is served by Amazon RDS."

Deirdre Kirmis (Jul 07 2026 at 18:00):

I would help myself a lot more if I just read through the entire section in the guides.

Philip Durbin 🚀 (Jul 07 2026 at 18:06):

I mean, it hasn't been updated it a while, but please be my guest! :sweat_smile:

Deirdre Kirmis (Jul 07 2026 at 18:23):

That whole Preparation page is really helpful. As is Jim's scaling tips page.

Deirdre Kirmis (Jul 07 2026 at 18:27):

Does Harvard run community-dev'd applications (ie: metrics, globus transfer, rdm dash, python scripts, etc) .. on a separate server from these, or on the existing web server(s)?

Philip Durbin 🚀 (Jul 07 2026 at 18:30):

Separate. The servers mentioned above are dedicated to production.

Thomas van Erven (Jul 09 2026 at 07:34):

New instance I'm setting up will run on Nomad with containers with configuration/microProfile as part of Nomad Vault.

Thomas van Erven (Jul 09 2026 at 08:10):

@Philip Durbin 🚀 did you lads ever run your containers on Fargate and if so, what was the experience?

Philip Durbin 🚀 (Jul 09 2026 at 10:53):

If we're using Fargate, I'm not aware of it. @Leo Andreev and @Steven Winship would know.

Deirdre Kirmis (Jul 10 2026 at 14:24):

@Thomas van Erven if you get things running on Fargate I would be very interested as we are moving many of our services for other apps there (ECS). Even if we could just run SOLR there that would be cool.

Deirdre Kirmis (Jul 10 2026 at 19:04):

Actually, thinking more about it .. if we could run SOLR in ECS with the current installation that would be immense. We are running it on a separate instance now, so shouldn't that work? Going to see if I can make that happen. We eventually do want to go to the container installation, but for now that would be helpful.

Last updated: Aug 18 2026 at 08:27 UTC