Ash Manda (Sep 24 2026 at 15:09): Ash Manda (Sep 24 2026 at 15:09):Hello. I am trying to do an installation of Dataverse from scratch using the official guides. I will note my observations here as I go through the motions.
Ash Manda (Sep 24 2026 at 15:10):
It appears that this recommendation is outdated: https://guides.dataverse.org/en/latest/installation/prerequisites.html
That specific setting does not exist in the domain.xml anymore
Ash Manda (Sep 24 2026 at 15:12):It should also be noted for any developers who use tools like mise-en-place (maybe just me?), that the install process does not play well with shims, and the standard dnf java install is highly preferred
Ash Manda (Sep 24 2026 at 15:15):The official recommendation is still PSQL 16, which is what I'll be doing to see if I'm hitting any issues
Ash Manda (Sep 24 2026 at 15:24):
There's no context behind this statement, I would've skipped it if not for a discussion with Gemini. I only had to change "scram-sha-256" to "md5"
Ash Manda (Sep 24 2026 at 15:28):There is a small mention about the installer using postgres pwd auth, but by default postgres does not use pwd auth. Maybe it would make sense to add a step about creating a password. If the installer truly does require a password, this might confuse every person trying to install the software
sudo -u postgres psql
ALTER USER postgres WITH PASSWORD 'your_secure_password';
\q
Philip Durbin 🚀 (Sep 24 2026 at 15:42):This is all great stuff! Thanks, @Ash Manda! 
While you're in there, maybe you can tell us if we still need these lines I'm highlighting:
We just discussed them at standup and put it on the tech hours agenda to ask @Leo Andreev and @Don Sizemore if we still need them.
Ash Manda (Sep 24 2026 at 16:47):I am not entirely sure. We defo need the chown dataverse line, but I'm not sure if we chown the lib back to root. I ran chown dataverse /usr/local/payara7 fully instead of just lib because I was running into some odd permissions issues
Ash Manda (Sep 24 2026 at 17:01):
Useradd does not create a password by default. We want to run "su - solr" to become the solr user, but it is not possible to log into that user unless you run "sudo passwd solr" and set a password beforehand. Or we want to point at "sudo -u solr -i"
Ash Manda (Sep 24 2026 at 17:08):
Although it seems blindingly obvious, there is no mention of needing to download a release previously in the guide within the prereq section. So if I was installing this from scratch, right during the solr step I would think "Wait, I needed a release downloaded already? Do I download it now? Do I do it as the solr user which I am currently logged into? Or should I exit?"
Ash Manda (Sep 24 2026 at 17:31):https://archive.apache.org/dist/solr/solr/9.8.0/solr-9.8.0.tgz
This link for downloading solr seems to be severely throttled, never rising above 200KB/s while I'm on a 3 Gigabit connection according to fast.com
That's about 30mins to an hour just to download this file. If at all possible, we might want to look into pointing at high speed mirrors
Don Sizemore (Sep 24 2026 at 17:47):@Ash Manda on Solr. Dataverse only uses certain versions of Solr, and mirrors only mirror the most/more recent major/minor versions. So we use archive.apache.org, which is indeed throttled.
Ash Manda (Sep 24 2026 at 17:49):We can build a free bypass using Cloudflare R2 and a SHA file if we need to. But, thanks for the comment! I was just confused if the terminal was frozen.
Philip Durbin 🚀 (Sep 24 2026 at 17:51):Ash Manda said:
there is no mention of needing to download a release previously in the guide within the prereq section
Hmm, good point. How about this change?
OLD: You should already have a “dvinstall.zip” file that you downloaded from https://github.com/IQSS/dataverse/releases .
NEW: Download “dvinstall.zip” from https://github.com/IQSS/dataverse/releases .
Ash Manda (Sep 24 2026 at 17:52):it should happen before we switch into the solr user, yes? So right before that step would be good?
Philip Durbin 🚀 (Sep 24 2026 at 17:57):No strong preference. It's not the end of the world if the solr user unzips dvinstall.zip into /tmp.
Ash Manda (Sep 24 2026 at 18:07):I was confused looking at the phrasing here. /etc/security/limits.conf is a readonly file that requires sudo to write to. the solr user is not a sudoer. So does this mean that the init script takes care of this part automatically? Or should I switch from solr to root and switch back to solr? There doesn't appear to be guidance on when to be solr, when not to, or if it even matters.
Ash Manda (Sep 24 2026 at 18:11):I also remember being told or seeing somewhere that sudo/root privileges will not be required to install Dataverse, but at least in my personal experience on Rocky 10, this has not been the case. Two best examples are the useradd command and this limits.conf file. Maybe its worth noting this somewhere in case the person trying to install the software doesn't have sudo privs on the machine? Or maybe it possible to go around these without sudo too? Just something I observed during the process
Philip Durbin 🚀 (Sep 24 2026 at 18:21):The nproc stuff was added in https://github.com/IQSS/dataverse/commit/d5df4882be283da11426386ca25c354ce3728203 by @Don Sizemore so let's ask him. :smile:
Philip Durbin 🚀 (Sep 24 2026 at 18:24):Hmm, maybe under https://guides.dataverse.org/en/6.12/installation/prerequisites.html#linux we can note that root access is required for certain operations such as creating service users.
Ash Manda (Sep 24 2026 at 18:43):
These commands will not be running at this point as it requires solr.service to be running systemd or otherwise. It might make sense to move this section to the part after when solr is already running
Philip Durbin 🚀 (Sep 24 2026 at 18:52):Sure, sounds good.
Philip Durbin 🚀 (Sep 24 2026 at 18:52):Thanks again for doing this exercise! :sweat_smile:
Philip Durbin 🚀 (Sep 24 2026 at 18:53):What's your overall impression so far? That installing Dataverse is overwhelming??! :grimacing:
Ash Manda (Sep 24 2026 at 18:53):The docs are surprisingly well written. Kudos to the entire team
Ash Manda (Sep 24 2026 at 18:54):These are minor friction points that I just wanted to note and put out there. On a normal day, I would just blink past them
Philip Durbin 🚀 (Sep 24 2026 at 18:55):Ok, phew. I must admit that I haven't run through them in a LOOONG time. When I need a Dataverse server, I use dataverse-ansible.
Ash Manda (Sep 24 2026 at 18:57):Speaks to how good dataverse-ansible has become!
Philip Durbin 🚀 (Sep 24 2026 at 18:58):Now nobody has to read that wall of text you're reading! :crazy:
Ash Manda (Sep 24 2026 at 19:13):The installing R section requires some prereqs that don't appear in the doc and will fail for anyone trying to set up R. I would consider this a mission-critical path as tabular ingest for .R files will not go through without R installed. We have hit this issue before where not having R installed would block all kinds of uploads that included an R file. This may cause the installation to fail silently.
I am going to include some commands that helped me
Ash Manda (Sep 24 2026 at 19:15):sudo dnf install -y \
gcc \
gcc-c++ \
make \
cmake \
autoconf \
automake \
openssl-devel \
libcurl-devel \
libxml2-devel \
zlib-devel \
bzip2-devel \
xz-devel \
R-core \
R-core-devel
This is the list of prereqs you would require to install the R libraries. Usually on a normal dev system, this would be masked by sudo dnf groupinstall "Development Tools"
Ash Manda (Sep 24 2026 at 19:16):Command to install the packages:
sudo Rscript -e '
pkgs <- c("R2HTML", "rjson", "DescTools", "Rserve", "haven")
install.packages(
pkgs,
repos = "https://cloud.r-project.org/",
lib = "/usr/lib64/R/library",
Ncpus = parallel::detectCores()
)
'
Command to test the packages installed:
Rscript -e '
pkgs <- c("R2HTML", "rjson", "DescTools", "Rserve", "haven")
results <- sapply(pkgs, require, character.only = TRUE, quietly = TRUE)
print(results)
if (!all(results)) {
cat("\nFailed to load:", names(results)[!results], "\n")
quit(status = 1)
} else {
cat("\nAll R packages verified successfully.\n")
}
'
Philip Durbin 🚀 (Sep 24 2026 at 19:30):Ash Manda said:
mission-critical path as tabular ingest for .R files will not go through without R installed.
Not everyone shares UNC enthusiasm for ingesting .RData files. :smile: Please see #7249. But sure, let's fix the docs for the people who want to keep it working!
Ash Manda (Sep 28 2026 at 14:27):I am going to resume this process today, but skipping all of the R-installation steps. We might want to to remove that section?
Philip Durbin 🚀 (Sep 28 2026 at 14:29):Hmm. Let's talk to @Leo Andreev about it first before removing it! I do feel like we should do something with that section to set expectations. Maybe a note at the top, at least. :thinking:
Please remind me, does the R install work inside of containers? Do the API tests exercise the feature of converting to and from RData format?
Ash Manda (Sep 28 2026 at 14:34):If its disabled by default, which I think it is, they shouldn't be exercising that feature, no, but I think its worth asking Oliver
Philip Durbin 🚀 (Sep 28 2026 at 14:35):Ok, thanks. For the purpose of going through the how to install docs, I agree with skipping the R stuff for now.
Ash Manda (Sep 28 2026 at 14:42):
At the moment, running sudo dnf install python installs python 3.12 on Rocky10. If Python 3.9 is a hard requirement for counter-processor then anyone running Rocky 10 would have to compile 3.9 from scratch. It might be worthwhile making a note about this somewhere as it might encourage people to choose Rocky 9 where the default would be python 3.9 or use mise-en-place or other such tools. I am proceeding with 3.12 to see if I run into any issues
Philip Durbin 🚀 (Sep 28 2026 at 14:45):Ok. @Steven Winship do you have any insight on which Python version one should use with Counter Processor?
Ash Manda (Sep 28 2026 at 14:48):
For anyone going through these docs, by this point through the solr install step, they should already have dvinstall.zip and the dvinstall folders within their "/tmp" directory. This could give them context and prevent them from redownloading and possibly causing drift within their systems by reusing the same dvinstall they used for solr
Philip Durbin 🚀 (Sep 28 2026 at 14:54):Yeah, I was wondering about that, from the conversation above. :sweat_smile:
Ash Manda (Sep 28 2026 at 14:55):
I am of the opinion that this paragraph is unnecessarily condescending. There are a few simple steps to determine whether postgres is installed or not in the system.
If I was a new user reading this paragraph, I would be highly tempted to not ask any question in the first place in fear of being called stupid.
A simple diagnostic saying try X, Y, Z first before reaching out to us gets the job done.
Steven Winship (Sep 28 2026 at 15:01):Interestingly I am in the process of installing counter_processor on QA, which is 8. I am compiling python 3.10 now. The Documentation says that 3.10 is required and I have had difficulty setting everything up with a lower version. Prod is running the lower version but I can’t remember who got that working.
Ash Manda (Sep 28 2026 at 15:05):Maybe I could try that specific path one day and see if its possible to get it working
Ash Manda (Sep 28 2026 at 15:06):Ash Manda said:
I am going to resume this process today, but skipping all of the R-installation steps. We might want to to remove that section?
In response to this, if Rserve documentation is altered in some way, it should probably reflect in the dataverse installer too as it asks for that information
Ash Manda (Sep 28 2026 at 15:09):The guides did not mention needing a relay or a mock SMTP server running on localhost before running the installer. I am going to set up postfix. We might want to add that in prereq?
William Hahn (Sep 28 2026 at 15:13):Ash Manda said:
The guides did not mention needing a relay or a mock SMTP server running on localhost before running the installer. I am going to set up postfix. We might want to add that in prereq?
in the wild, I don't think people will be running it on localhost, but could be worthwhile to say that users take note of it at some capacity (their provider / org's mail server)
Ash Manda (Sep 28 2026 at 15:20):PER01003: Deployment encountered SQL Exceptions:
PER01000: Got SQLException executing statement "CREATE TABLE EXTERNALTOOLTYPE (ID BIGINT SERIAL NOT NULL, TYPE VARCHAR(255) NOT NULL, EXTERNALTOOL_ID BIGINT NOT NULL, PRIMARY KEY (ID))": org.postgresql.util.PSQLException: ERROR: syntax error at or near "SERIAL"
Ash Manda (Sep 28 2026 at 15:21):So I'm assuming PQSL 16 is a no-go
William Hahn (Sep 28 2026 at 15:21):That confirms it; you will definitely need 17!
William Hahn (Sep 28 2026 at 15:24):William Hahn said:
Ash Manda said:
The guides did not mention needing a relay or a mock SMTP server running on localhost before running the installer. I am going to set up postfix. We might want to add that in prereq?
in the wild, I don't think people will be running it on localhost, but could be worthwhile to say that users take note of it at some capacity (their provider / org's mail server)
to clarify, I believe localhost is fine if you're running postfix etc. and pointing to the correct mail relay there
Philip Durbin 🚀 (Sep 28 2026 at 15:28):Ash Manda said:
I am of the opinion that this paragraph is unnecessarily condescending... If I was a new user reading this paragraph, I would be highly tempted to not ask any question in the first place in fear of being called stupid.
Yeah, I agree. Let's edit it!
Philip Durbin 🚀 (Sep 28 2026 at 15:31):William Hahn said:
That confirms it; you will definitely need 17!
Right. We are updating the docs in this PR @Don Sizemore made: recommend PG17 as an immediate fix for new installations - #12738
William Hahn (Sep 28 2026 at 16:49):We wanted @Ash Manda to try out 16 first, just to see what would happen :sweat_smile:
Same thing as what I hit running dataverse-ansible
Philip Durbin 🚀 (Sep 28 2026 at 16:50):Such a bummer but it is what it is, I guess.
William Hahn (Sep 28 2026 at 16:52):to be fair, I'm wondering if it will work again once https://github.com/payara/Payara/issues/8250 is closed
Philip Durbin 🚀 (Sep 28 2026 at 16:53):Yep, we'll keep an eye on that issue. :eyes:
Ash Manda (Sep 28 2026 at 16:58):
Ripping out PSQL 16, and placing in PSQL 17 did the job
Ash Manda (Sep 28 2026 at 17:00):This concludes my manual Dataverse Installation process
Ash Manda (Sep 28 2026 at 17:02):Overall my impression is that the docs and installers are set up surprisingly well. Especially kudos to whoever worked on the install.py script, it was fantastic, so easy to run. But, I kept running into some or the other odd permissions issues while going through this. I ended up buckling and running the install.py script as root because the dataverse user seemed to be lacking permissions which is odd because I followed every line dot to dot. So there might be some permission gremlins the team might eventually want to iron out within the docs.
William Hahn (Sep 28 2026 at 17:09):So, the issue was mainly ownership? Or were there file permissions problems too?
Ash Manda (Sep 28 2026 at 17:11):There were both. I tried running install.py as dataverse and the script executed successfully and then when opening localhost:8080 it ran into a 500 Server Error saying dataverse release is null. I had to stop payara, drop the db, and the user. Then I restarted the process with sudo and everything went through flawlessly. I am assuming that the dataverse user was missing certain permissions to write to certain files and launch certain processes.
Don Sizemore (Sep 28 2026 at 17:24):There should be an as-setup.out alongside the install script, and Dataverse-Ansible (if not the installer?) writes an install.out as well. I'm curious what permissions problems you ran into.
Don Sizemore (Sep 28 2026 at 17:25):We do _not_ want to run Payara as root. (or Solr, or Postgres, or anything else)
Ash Manda (Sep 28 2026 at 17:40):there's no .out file I see in the folder. And I ran just the installer as root, everything else runs on its own user. Payara as dataverse, sol as solr, Postgres as dvnapp and postgres
But I remember seeing jhove permission issue stuff along with some other things. It could very well be that I missed something, but I doubt it. I followed the docs pretty blindly though
William Hahn (Sep 28 2026 at 17:46):You probably already checked this, but according to the install steps it will fail if the user lacks write to any of the following
- /usr/local/payara7/glassfish/lib
- /usr/local/payara7/glassfish/domains/domain1
- the current working directory of the installer (it currently writes its logfile there), and
- your jvm-option specified files.dir
yep I gave dataverse chown over the entire /usr/local/payara7 directory as I mentioned earlier in the conversation, and over /tmp (which is where the dvinstall was) All the jvm-options were created just fine so I don't think it was that
Philip Durbin 🚀 (Sep 28 2026 at 18:54):Ash Manda said:
Overall my impression is that the docs and installers are set up surprisingly well. Especially kudos to whoever worked on the install.py script, it was fantastic, so easy to run.
We have @Leo Andreev to thank for install.py!
Philip Durbin 🚀 (Sep 28 2026 at 18:55):Ash Manda said:
I ended up buckling and running the install.py script as root
Nooooo. :sweat_smile: This shouldn't be necessary.
Steven Winship (Sep 28 2026 at 19:53):Correction. Prod is running python3.11. I’m currently installing 3.11 on QA since 3.10 was giving me issues
Philip Durbin 🚀 (Sep 29 2026 at 15:54):Philip Durbin 🚀 said:
We just discussed them at standup and put it on the tech hours agenda to ask Leo Andreev and Don Sizemore if we still need them.
We just discussed this at tech hours. The highlighted lines can be removed.
Don Sizemore (Sep 29 2026 at 19:37):@Philip Durbin 🚀 if y'all find it friendly, I'd like to start pecking away at docs PRs while I'm a captive audience during the work day, which will free @Ash Manda up to continue his test automation work.
Philip Durbin 🚀 (Sep 29 2026 at 20:15):@Don Sizemore sounds great! Strike while the iron is hot! :fire:
Last updated: Nov 02 2026 at 07:01 UTC