Deirdre Kirmis (Oct 16 2023 at 15:23): Deirdre Kirmis (Oct 16 2023 at 15:23):Hi all .. recently I upgraded our QA site to v6.0 following the given instructions for java, payara, and solr. We have Dataverse running on an application server, and Solr and Rserve running on a separate server in AWS. All went well, it seemed, solr was running the new version and I could see all the datasets in Dataverse. Then, I restarted the Solr server, and now I'm getting a "connection refused" on the Dataverse landing page. I've seen this before, but I don't remember how to fix it. I verified that my solr.service isn't using the “-D jetty.host=127.0.0.1" option and made sure to upgrade java on the services server to v17. I see the server listening on port 8983 but don't see any connections. Any ideas what I missed?
Philip Durbin 🚀 (Oct 16 2023 at 15:24):Do you see fields on the equivalent of http://localhost:8983/solr/collection1/schema/fields ?
Deirdre Kirmis (Oct 16 2023 at 15:26):yes, but "curl http://localhost:8080/api/info/version" doesn't work
Philip Durbin 🚀 (Oct 16 2023 at 15:28):Huh. /info/version doesn't work but you're seeing a Solr error on the collection/home page?
Deirdre Kirmis (Oct 16 2023 at 15:33):oops sorry, ignore that .. i copied the wrong command .. this one works "curl http://localhost:8983/solr/collection1/schema/fields"
Deirdre Kirmis (Oct 16 2023 at 15:33):but still getting "Server refused connection at: http://<IP address>:8983/solr/collection1"
Deirdre Kirmis (Oct 16 2023 at 15:34):on the webserver
Philip Durbin 🚀 (Oct 16 2023 at 15:37):You're doing that "fields" curl test from the Payara server, right? Across the wire to the Solr server?
Deirdre Kirmis (Oct 16 2023 at 15:38):nope .. doesn't work on the web server
Deirdre Kirmis (Oct 16 2023 at 15:38):connection refused
Philip Durbin 🚀 (Oct 16 2023 at 15:40):Is there a firewall blocking it?
Deirdre Kirmis (Oct 16 2023 at 15:42):there shouldn't be .. i just upgraded the currently running instance .. and the current security group allows that port .. i didn't change that .. and didn't install any other firewall or add any rules
Deirdre Kirmis (Oct 16 2023 at 15:44):SolrHostColonPort is set correctly still
Philip Durbin 🚀 (Oct 16 2023 at 15:45):I'd stick with curl. If you can't curl the fields from the Payara server to the Solr server, we should fix that first. This has nothing to do with Dataverse.
Deirdre Kirmis (Oct 16 2023 at 15:47):ah so maybe something actually went wrong with the payara install?
Philip Durbin 🚀 (Oct 16 2023 at 15:54):Sorry, no, I'm saying it's probably a firewall.
Deirdre Kirmis (Oct 16 2023 at 15:56):okay checking all that out again .. my security groups seem fine .. server IPs didn't change .. if i change it back to old solr it works
Philip Durbin 🚀 (Oct 16 2023 at 15:58):Huh, old Solr works...
Deirdre Kirmis (Oct 16 2023 at 16:02):no now it isn't .. it did last night .. or if I change back to the volume where the old solr was running it works
Deirdre Kirmis (Oct 16 2023 at 16:02):going to revert the instances back to the old versions and try this again in a dev environment .. i must have messed something up
Deirdre Kirmis (Oct 16 2023 at 16:05):yea, so i reverted back to the volume (for the solr server) that had solr 8 running but kept the payara server the same (upgraded) and it works .. i am just detaching/attaching volumes and not changing any security groups
Deirdre Kirmis (Oct 16 2023 at 16:06):but, my webserver instance now is upgraded, but the solr instance is still running v8 .. will try the upgrade again
Deirdre Kirmis (Oct 16 2023 at 16:07):or . if it's okay to leave solr 8 running with upgraded payara and java .. will try the solr upgrade in a temp dev env
Deirdre Kirmis (Oct 16 2023 at 16:18):what's weird is .. after doing the upgrade initially .. it was running fine on new solr and the webserver was connecting .. but i must not have restarted the solr server after that .. that was a couple of weeks ago .. i restarted it this weekend, and after that it wouldn't connect
Deirdre Kirmis (Oct 19 2023 at 18:13):i created a dev environment and am trying this upgrade again .. at the end of the payara upgrade, i get a message that the deploy of dataverse-6.0.war failed .. however the site loads and shows that it is at v6.0 .. and the only errors that i see in the sql statements that display when upgrading are related to things already existing .. is this normal? is there a way for me to see why it failed? i can't remember if i've gotten a "failed" message on deploy before .. i do always see the "already exists" messages though
Philip Durbin 🚀 (Oct 19 2023 at 18:14):Yeah, unfortunately some of that noise is normal. :disappointed:
Philip Durbin 🚀 (Oct 19 2023 at 18:15):Since this is a dev env I assume it's safe for you to post the output here.
Deirdre Kirmis (Oct 19 2023 at 18:16):oh oops, i guess this time it really failed .. the first time i did this it said it failed but it actually deployed .. this time not :thinking:
Deirdre Kirmis (Oct 19 2023 at 18:16):oh i did it again and it worked
Deirdre Kirmis (Oct 19 2023 at 18:17):well, it says "Deploy completed with warnings" which is what i usually see .. but the site isn't coming up .. wow this upgrade is a bear
Deirdre Kirmis (Oct 19 2023 at 18:19):yea going to the site just shows payara admin page .. something messed up
Philip Durbin 🚀 (Oct 19 2023 at 18:20):It's like hitting the TV with a stick until it works.
Deirdre Kirmis (Oct 19 2023 at 18:28)::grinning_face_with_smiling_eyes:
Deirdre Kirmis (Oct 19 2023 at 18:28):haven't even tried the solr upgrade yet!
Philip Durbin 🚀 (Oct 19 2023 at 18:43):fun times ahead! :tada:
Deirdre Kirmis (Oct 19 2023 at 19:42):interesting, the payara site was showing because the load balancer registered my instance as unhealthy .. i must have taken too long to upgrade! :sweat_smile:
As for upgrading solr, one question that i have is .. the instructions say to su - as the solr user, but that user is unprivileged, so doesn't have permission to wget or unzip in /usr/local/solr, or to copy files as indicated in the instructions. Do you have your solr user in sudoers? Or special permissions? The main installation instructions say to chown to solr for /usr/local/solr but that is not recursive.
Philip Durbin 🚀 (Oct 19 2023 at 20:11):Huh. Not recursive? I thought it was.
Philip Durbin 🚀 (Oct 19 2023 at 20:13):I'm looking at this:
useradd solr
mkdir /usr/local/solr
chown solr:solr /usr/local/solr
From https://guides.dataverse.org/en/6.0/installation/prerequisites.html#installing-solr
Philip Durbin 🚀 (Oct 19 2023 at 20:14):So /usr/local/solr should be owned by the solr user. Right?
Deirdre Kirmis (Oct 19 2023 at 20:27):yes, but nothing under that directory is .. that would be chown -R solr:solr /usr/local/solr
Deirdre Kirmis (Oct 19 2023 at 20:28):i did the full upgrade again .. solr won't start .. eegads what am i doing wrong?
Deirdre Kirmis (Oct 19 2023 at 20:29):if i point the solr.service file back to the old solr (/usr/local/solr) it works fine using the old version
Philip Durbin 🚀 (Oct 19 2023 at 20:32):I would chown -R it to the solr user.
Deirdre Kirmis (Oct 19 2023 at 20:35):okay just wasn't sure if some things needed to stay as root .. going to do that and go through the steps again .. thanks!
Philip Durbin 🚀 (Oct 19 2023 at 20:37):Basically, Solr should be running as the solr user. So that user needs to own the files.
Philip Durbin 🚀 (Oct 19 2023 at 20:37):You know, a service user.
Deirdre Kirmis (Oct 19 2023 at 20:41):yea i thought so and actually tried changing the ownership of everything before but that still doesn't fix it .. solr still won't start .. back to banging the stick
Philip Durbin 🚀 (Oct 19 2023 at 20:41):heh
Deirdre Kirmis (Oct 19 2023 at 20:55):fixed .. deleted the solr-9.3.0 directory and started all over, after chown -R /usr/local/solr and doing the entire install as solr user and now it works .. i must not have done that correctly before although i thought i did .. omg .. sorry to bother you .. thanks for your help!
Philip Durbin 🚀 (Oct 19 2023 at 20:58):No problem! Should we fix the release notes?
Deirdre Kirmis (Oct 19 2023 at 20:59):agh .. and i restarted the server and now solr doesn't start! this is what happened originally .. i upgraded everything but didn't restart the server, and everything was running, but about 2 weeks later i restarted the server and solr wouldn't start .. well i know it CAN work
Deirdre Kirmis (Oct 19 2023 at 21:03):not sure about the release notes .. i'm really an ubuntu person .. just have rocky linux for this .. although i think it is the same, most folks probably know and don't follow each line so specifically :grinning_face_with_smiling_eyes:
Philip Durbin 🚀 (Oct 19 2023 at 21:10):The release notes could certainly be off. I wrote most of it. :sweat_smile:
Deirdre Kirmis (Oct 19 2023 at 21:12):the are great! immensely helpful!
Deirdre Kirmis (Oct 19 2023 at 21:14):i can't figure out why solr would start fine after upgrading, but not start after a reboot
Deirdre Kirmis (Oct 19 2023 at 22:49):well, it is working .. not sure what fixed it .. i just wiped out and reinstalled about 4 more times .. i restart the instance and solr service starts .. i can see my datasets and add new from the console, etc .. but the last weird thing is that .. although solr starts, and the reindex command works from the web server .. the curl command to show the schema fields still gets a connection refused error .. i have a SG rule in place allowing port 8983 from the webserver and i just added a rule to allow everything from the webserver .. idk
Philip Durbin 🚀 (Oct 20 2023 at 15:58):Yeah, we've got to get that curl command working. You can do it! 
Deirdre Kirmis (Oct 20 2023 at 19:53):if anyone else deals with this .. @Don Sizemore found an alert in the solr docs indicating that "Solr now binds to localhost network interface by default for better out of the box security." We have to add our server IP to the "SOLR JETTY HOST" to allow external solr servers. YAY THANK YOU!!!! I was pretty focused on my installation being wrong and didn't consider actually looking at the solr docs.
Philip Durbin 🚀 (Oct 20 2023 at 19:54):Yes, I noticed he just opened https://github.com/GlobalDataverseCommunityConsortium/dataverse-ansible/issues/328 about removing jetty.host in the future.
Don Sizemore (Oct 20 2023 at 20:04):Philip Durbin said:
Yes, I noticed he just opened https://github.com/GlobalDataverseCommunityConsortium/dataverse-ansible/issues/328 about removing jetty.host in the future.
I also opened number 10030! Will submit PR.
Philip Durbin 🚀 (Oct 20 2023 at 20:06):/me looks at #10030
Philip Durbin 🚀 (Oct 20 2023 at 20:06):nice
Deirdre Kirmis (Oct 20 2023 at 20:07):Don, thanks again. You two are so immensely helpful to me (and everyone else). As I've said often before, ASU wouldn't have a dataverse installation without you all!
Philip Durbin 🚀 (Oct 20 2023 at 20:07):Aw. Such a nice way to go into the weekend. Thanks, Deirdre.
Deirdre Kirmis (Oct 20 2023 at 20:12)::glowing_star:
Philip Durbin 🚀 (Oct 24 2023 at 18:18):@Don Sizemore just made a pull request for the issue above: document localhost-only behavior of Solr-9.3.0 and later #10037 (thanks!)
Last updated: Oct 30 2025 at 06:21 UTC