Problem uploading large file with python dvuploader · troubleshooting

I installed a new dataverse server for testing purposes. It is running dataverse 6.9 and was installed using the dataverse ansible from GDCC. I created a random 30GB files and tried to upload to an empty dataset using python dvuploader. After arriving to 100%, the process stalls forever and doesn't give a clue about what is happening. The machine has enough disc and RAM to handle that file.

When the process arrives 100%, the files appears at /tmp as "MIME0000000000.tmp". I guess that is the ingestion process running. But after sometime, that files dissapears and I can't find any 30GB files in the system. In the apache logs I can see only this related to this upload (the DOI is the fake one for testing purposes):

I tried to upload the same file to my production dataverse server, which is running still v6.5 and had no issue on the process.

Carlos Perez-Miguel (Jun 17 2026 at 14:47):

After some more digging, I found in the logs of payara this:
java.io.IOException: No space left on device

This server was a VM on proxmox with a 30 gb disk and an attached NFS storage for the datasets with plenty of space. It is strange because the system was configured to use a folder on this NFS storage for uploading purposes by using the JVM option -Ddataverse.files.uploads, but it seems that somehow, first the system tries to upload the file to the root filesystem and then moves it to the NFS storage.

I am recreating the VM with more space. Does someone know what more jvm options could apply in this case?

Philip Durbin 🚀 (Jun 17 2026 at 14:56):

Don Sizemore (Jun 17 2026 at 17:37):

@Carlos Perez-Miguel Depending on the filetype, Dataverse may attempt to make another copy beneath /usr/local/payara7/glassfish/domains/domain1/generated , if the root filesystem didn't have room for 2 copies of each file before copying to the NFS mount, that could explain the "no space left on device" error you're seeing. What filename extension did you use?

Carlos Perez-Miguel (Jun 18 2026 at 07:07):

@Don Sizemore the extension was ".txt". That file under the "generated" folder is removed after the ingestion is made or rests in there?

Carlos Perez-Miguel (Jun 18 2026 at 08:32):

New test:
Now the VM has 256 GB of disk and 64 GB of RAM. Plenty of space for the file. Also, the final filesystem for datasets is the NFS folder. The file to upload is 3GB size of random data with the name sample_3gb.txt.

Dvuploader uploads once but then the file appears several times on the /tmp folder. Each copy has the name MIMEXXXXXXXX.tmp. Sometimes the files disappears and the system creates a new copy

Exception shown on the payara log:
Could not find key "externaltools.textPreviewer.displayname" in bundle file: |#]
And then:

Caused by: Exception [EclipseLink-5010] (Eclipse Persistence Services - 4.0.1.payara-p3.v202409231228): org.eclipse.persistence.exceptions.OptimisticLockException
Exception Description: The object [[DatasetVersion id:38]] cannot be merged because it has changed or been deleted since it was last read.

And then more:
This file has the same content as sample_3GB.txt that is in the dataset. |#]

[#|2026-06-18T08:02:59.746+0000|INFO|Payara 6.2025.3|edu.harvard.iq.dataverse.ingest.IngestServiceBean|_ThreadID=104;_ThreadName=http-thread-pool::jk-connector(4);_TimeMillis=1781769779746;_LevelValue=800;|
  Incrementing recorded storage use by 3221225472 bytes for dataset 6139|#]

After these errors, this 3GB file appears several times on the Dataverse UI with variations the original name but the dvuploader process never ends:

sample_3GB.txt
sample_3GB-1.txt
sample_3GB-2.txt

It seems that the system enter in some kind of loop trying to process the txt file, fails somehow, rinse and repeat.
I tried to upload it with the web UI and it doesn't finish but no files appear anywhere.

I also tried the same thing with a 30MB random txt file and it works perfectly with both dvuploader and the web UI.

Philip Durbin 🚀 (Jun 18 2026 at 09:30):

Philip Durbin 🚀 (Jun 18 2026 at 10:20):

sample_3GB.txt
sample_3GB-1.txt
sample_3GB-2.txt

That is, when duplicates are uploaded, it's normal that -1, etc. is added to the filename.

Carlos Perez-Miguel (Jun 18 2026 at 10:37):

But those 3 were created during the same upload session while Dvuploader was at 100%. I'll try the same with more versions of Dataverse and will create an issue eventually. Thank you @Philip Durbin 🚀

Philip Durbin 🚀 (Jun 18 2026 at 10:55):

Carlos Perez-Miguel (Jun 19 2026 at 09:16):

Philip Durbin 🚀 (Jun 19 2026 at 10:42):

Jan Range (Jun 19 2026 at 11:12):

@Philip Durbin 🚀 thanks for tagging me! The upload seems to stream the file content successfully, since the progress bar is tied to a stream handler. My guess is that the REST API 500error keeps it from completing. It could be that the retry strategy fires and the file is uploaded again, while the progress bar is not reset.

When you upload the file again, do you spot that the self-hosted instance suddenly restarts? This has been an issue before that I was able to reproduce, but havent yet found a reason for why it is happening.

Carlos Perez-Miguel (Jun 22 2026 at 08:55):

@Jan Range , after more tests with both the REST API and python dvuploader, I can say that it fails after the server answers with a 500 status code. In any case, I don't see any restart of the payara service.

Stream: troubleshooting

Topic: Problem uploading large file with python dvuploader

Carlos Perez-Miguel (Jun 17 2026 at 14:20):