Bethany Seeger (Sep 02 2026 at 21:10): Bethany Seeger (Sep 02 2026 at 21:10):We have a large dataset with ~150,000 files uploaded so far and counting. It is currently still in draft form.
When I look at it, it takes 30 s for the page to load. We thought maybe Solr was a bottle neck, so we gave the system more RAM (Solr has 12 G and DV has 20 G). That didn't speed up or improve things at all and the dataverse app might actually be in worse shape now.
Not sure what's going on behind the scenes, but is Dataverse loading all the information about all the files in the dataset when it's viewed?
Has anyone else dealt with this? Any suggestions on how to tune the server to work with this?
Philip Durbin 🚀 (Sep 03 2026 at 11:26):I'm happy to hear what others think but in the guides we suggest avoiding many files: https://guides.dataverse.org/en/6.11/admin/big-data-administration.html#avoiding-many-files
The solutions I'm aware of:
Bethany Seeger (Sep 03 2026 at 13:05):This makes sense - though it sounds like DV loads info about the whole set from solr into memory -- perhaps that could be improved upon, with pagination?
In the mean time it sounds like we should try to get this set split up. Is it possible to move a file (I'm not see docs on it in my quick search, nor any options in the UI)? The user has spent weeks uploading this dataset, so I'd feel bad if our solution was to suggest they re-upload it into different datasets.
Thanks, Phillip!
Philip Durbin 🚀 (Sep 03 2026 at 13:15):Yeah, it's a bummer if so much time has already been spent.
No, there's no way to move a file from one dataset to another.
You could script something, I guess.
Philip Durbin 🚀 (Sep 03 2026 at 13:17):The new UI has infinite scroll of data files: https://github.com/IQSS/dataverse-frontend/pull/415
Last updated: Oct 02 2026 at 18:51 UTC