Stream: troubleshooting

Topic: HTTP 500 during multipart upload finalization (~25 GB file)

view this post on Zulip Manaswini Ganjam (Sep 02 2026 at 23:45):

Hi everyone,

I am uploading a large dataset to Harvard Dataverse using DVUploader v1.4.0 in direct-upload mode from a university HPC data transfer node.

I am encountering an HTTP 500 error during the finalization of a multipart upload, and I am hoping someone familiar with the Dataverse/S3 upload pipeline can help determine whether this is a known issue.

The upload proceeds normally through the S3 multipart stage:

  2. followed by: “Not uploaded due to error during processing.”

Unfortunately, DVUploader does not appear to log the response body from the HTTP 500, so I do not have the server-side exception.

I previously encountered a different error (SignatureDoesNotMatch) during another large-file upload, but the current failure is specifically an HTTP 500 after all multipart parts have successfully uploaded.

I am particularly interested in whether anyone has encountered this sequence:

S3 multipart parts successfully uploaded → Dataverse multipart completion/finalization → HTTP 500 → file not registered

and whether there are known limitations related to file size (~25 GB), number of multipart parts (25), or the Dataverse finalization/registration step.

I have approximately 39 archives, averaging ~24 GB each to upload. If anyone has suggestions for additional logging, relevant Dataverse configuration, known issues, or a GitHub issue that matches this behavior, I would greatly appreciate the guidance.

Thank you,
Manaswini

view this post on Zulip Philip Durbin 🚀 (Sep 03 2026 at 11:30):

@Manaswini Ganjam hi! Thanks for all the details. Can you please send them to support@dataverse.harvard.edu? This will generate a ticket and once the problem is solved we can summarize here in Zulip.

view this post on Zulip Philip Durbin 🚀 (Sep 03 2026 at 13:27):

Oh good, you also posted to https://groups.google.com/g/dataverse-community/c/YNy5G8vlGAg/m/6ZzqkqP3BQAJ

view this post on Zulip Manaswini Ganjam (Sep 03 2026 at 16:55):

@Philip Durbin 🚀 Hi, I submitted a ticket and the response is below.
Response: Hi Manaswini,
Yes, I understand the problem with dv-uploader. Let me see if someone can look
at this when they have time. No guarantees, since the team is in the middle of
a release. You may want to connect with the Google community or the Dataverse
Zulip community and ask your question there and see if anyone has encountered
the same problem before.

What I am saying is that, since the project is so large, you would not have to
deal with DV uploader issues if you used the paid storage option. It's $11/TB
per year, plus 30% overhead on the total price. Minimum 6 years of storage. But
if you would rather wait for a solution, that is completely an option.
Regards

view this post on Zulip Philip Durbin 🚀 (Sep 03 2026 at 18:08):

Ah, in https://help.hmdc.harvard.edu/Ticket/Display.html?id=423045 . Thanks.

As stated, we're trying to get a release out. :sweat_smile: See #community > Release 6.12 Timeline

What's your take on the responses you've gotten so far at https://groups.google.com/g/dataverse-community/c/YNy5G8vlGAg/m/6ZzqkqP3BQAJ ?

view this post on Zulip Manaswini Ganjam (Sep 03 2026 at 19:09):

I believe the responses are suggesting admin-side config change (DataFileChecksumValidationSizeLimit + disabling FileValidationOnPublishEnabled), and checksum computation on a 25 GB file finalizing server-side is a very plausible timeout culprit for an HTTP 500 that happens right at completion. But I will need Harvard's Dataverse admins to apply it. I submitted the intake form for large data.

view this post on Zulip Philip Durbin 🚀 (Sep 04 2026 at 13:52):

Right, hmm. I think it's unlikely we'll turn off FileValidationOnPublishEnabled on Harvard Dataverse. I'm not sure if we have DataFileChecksumValidationSizeLimit set or not. The person would who advise on the technical side is on vacation, it being Labor Day weekend and all. We he gets back I'll ask him to look at your ticket. Meanwhile, thanks for submitting the large data intake form. (The person who reads those is also on vacation, though.)

view this post on Zulip Philip Durbin 🚀 (Sep 04 2026 at 14:22):

Oh, I see Jim replied as well: https://groups.google.com/g/dataverse-community/c/YNy5G8vlGAg/m/4fgJJ9E_BgAJ

He's the author of DVUploader and worked a lot on this part of the backend as well.

Last updated: Oct 02 2026 at 18:51 UTC