Sherry Lake (Nov 13 2024 at 18:15): Sherry Lake (Nov 13 2024 at 18:15):There was some discussion of this back in February:
https://groups.google.com/g/dataverse-community/c/ZJRAuy-1PKE/m/bdIqRh5WAAAJ
When we were at V 5.14, I could use DVUploader on any size file. Now that we are at 6.2, it looks like the MaxFileUploadSize is now checked?
Can someone confirm?
Then if I have 6GB set (mainly for the UI), I will have to increase the :MaxFileUploadSizeInBytes to use DVUploader on "large" files?
Since DVUploader was touted as how to upload large files (and large amounts of files), I thought that there might be a different setting for DVUploader (or no checks).... but I know it is just using Native API commands.
Any thoughts on having two "Max" upload sizes?
Philip Durbin 🚀 (Nov 13 2024 at 18:26):Interesting. It looks like back then I pointed at https://github.com/IQSS/dataverse/blob/v6.1/src/main/java/edu/harvard/iq/dataverse/ingest/IngestServiceBean.java#L361 ...
if (fileSizeLimit == null || confirmedFileSize < fileSizeLimit)
... added in this PR: Collection Quotas #10144
Philip Durbin 🚀 (Nov 13 2024 at 18:26):So I'm pretty confident that file size isn't being checked.
Philip Durbin 🚀 (Nov 13 2024 at 18:27):But you're saying you want some sort of mode where the file size is not checked? :thinking:
Philip Durbin 🚀 (Nov 13 2024 at 18:27):You want to have your cake and eat it too? :cake: :crazy:
Sherry Lake (Nov 13 2024 at 18:45):So you say "pretty confident that file size isn't being checked."
But actually, I think the file size is being checked now on DVUploader
With our Max File upload size set to 6GB, I got this error from DVUploader:
PROCESSING(F): D:\McCormickObs_MAIN_2024\McCormickObs_MAIN_2024_E57.e57
Does not yet exist on server.
Retrying request for file upload URL(s): return status was : 400
Retrying request for file upload URL(s): return status was : 400
Retrying request for file upload URL(s): return status was : 400
Retrying request for file upload URL(s): return status was : 400
Retrying request for file upload URL(s): return status was : 400
Not uploaded due to error during processing: D:\McCormickObs_MAIN_2024\McCormickObs_MAIN_2024_E57.e57Execution Complete.
When I set max upload size to 10GB (file to upload was 8GB).
It worked:
PROCESSING(F): D:\McCormickObs_MAIN_2024\McCormickObs_MAIN_2024_E57.e57
Does not yet exist on server.
Creating job for 1073741824 bytes
Creating job for 1073741824 bytes
Creating job for 1073741824 bytes
Creating job for 1073741824 bytes
Creating job for 1073741824 bytes
Creating job for 1073741824 bytes
Creating job for 1073741824 bytes
Creating job for 1073741824 bytes
All 8.0 parts for: s3://dataverse-storage-staging:19321b8dc03-41fddc523654 queued.
Philip Durbin 🚀 (Nov 13 2024 at 21:20):Sorry, sorry, I wrote that backwards. I'm pretty sure the file size IS being checked, as you've observed.
Philip Durbin 🚀 (Nov 13 2024 at 21:21):Is being checked as of that pull request, as of Dataverse 6.1.
Philip Durbin 🚀 (Nov 13 2024 at 21:22):But you want a way to override this somehow?
Sherry Lake (Nov 14 2024 at 12:04):As I said since DVUploader is touted as how to upload "large" files, I would like there to be a different maxSIze, IF uploading via api, that larger than the GUI limit.
Does this make sense @Philip Durbin 🐉 ? If so, shall I make an issue?
Philip Durbin 🚀 (Nov 14 2024 at 12:11):I mean, it doesn't really make sense to be. I like how it's a system-wide setting.
Why not just raise the limit? Does it result in a poor UI experience?
Sherry Lake (Nov 14 2024 at 12:56):My experience it does results in poor UI experience. The "software" gets in the way; I think. But that too might have changed with later versions.
Philip Durbin 🚀 (Nov 14 2024 at 16:24):I hope it's better with the new React frontend, the SPA. Have you tried https://beta.dataverse.org ?
Philip Durbin 🚀 (Nov 14 2024 at 16:24):Although maybe I should ask @Don Sizemore how much free disk space we have first. :sweat_smile:
Don Sizemore (Nov 14 2024 at 17:04):@Sherry Lake I love the idea of separate upload size limits depending on method (dv-uploader in particular). beta.dataverse.org currently has 12G available, but don't tell anybody.
Sherry Lake (Nov 15 2024 at 14:23):@Don Sizemore I will quietly :shush: test the SPA w/ a "large" file (well, < 12GB).
Philip Durbin 🚀 (Nov 15 2024 at 14:27):@Sherry Lake great. I can't wait to hear how it goes!
Sherry Lake (Nov 20 2024 at 15:32):Just getting around to comparing speed w/ beta.dataverse.org..... but it is "down", getting the "Hello from Payara" page.
@Don Sizemore ?? @Philip Durbin 🐉 ??
Philip Durbin 🚀 (Nov 20 2024 at 15:32):
Philip Durbin 🚀 (Nov 20 2024 at 15:33):Yeah, we know. We're working on it. Sorry!
Sherry Lake (Nov 20 2024 at 15:33):No prob. Just reporting, in case no one knew.
Philip Durbin 🚀 (Nov 20 2024 at 15:34):Once upon a time we had this issue:
Monitoring dashboard for shared resources use in dataverse #5760
Philip Durbin 🚀 (Nov 20 2024 at 15:37):@Sherry Lake we haven't heard from mbamouni in a while. Please feel free to create a fresh issue if you want some monitoring of beta, etc. Maybe @Juan Pablo Tosca Villanueva can add it to the hub. :grinning:
Last updated: Oct 30 2025 at 05:14 UTC