Stream: troubleshooting

Topic: ✔ Direct Upload (S3) and API "add" endpoint

Sherry Lake (Jul 09 2024 at 15:34):

Just a question trying to figure things out.

We have direct upload (to S3). If I have a "small" file ~ 300MB, can I still use the simple API "add" endpoint for the file,
OR
do I have to do the 3 steps as described on this page?
https://guides.dataverse.org/en/latest/developers/s3-direct-upload-api.html

Is there a size max, where the 3-step routine has to be used, or will the simple "add" use non-direct (can it do that)?

I use "dvuploader" so I don't have to think about the steps, but was wondering if my researchers would discover the simple "add" API first?

Philip Durbin 🚀 (Jul 09 2024 at 15:52):

I'm pretty sure you have to use the three step API.

Maybe the docs for the simple API should mention this.

Philip Durbin 🚀 (Jul 09 2024 at 15:52):

As you say, if you use the UI or DVUploader (and I believe dvwebloader), you don't have to think about it.

Philip Durbin 🚀 (Jul 09 2024 at 15:54):

What happens if you try the simply API? Does it tell you "sorry, you can't use this because direct upload is enabled?"

Sherry Lake (Jul 09 2024 at 16:15):

Oh.... you want me to test.... :stuck_out_tongue_wink:
I can do that.

Also the documentt does mention it, in the simple "add" section, it says:

Note that when a Dataverse installation is configured to use S3 storage with direct upload enabled, there is API support to send a file directly to S3. This is more complex and is described in the Direct DataFile Upload/Replace API guide.

but how will the uploader know that a particular installation is direct upload?

Philip Durbin 🚀 (Jul 09 2024 at 16:52):

Hmm, it must do some sort of check. I'm not sure.

Sherry Lake (Jul 12 2024 at 12:54):

Tested.....
When direct upload is configured - IF the file size is less than this JVM setting: dataverse.files.<id>.min-part-size
default is 1GB
THEN the simple API command works.

BUT if the file size is larger that the "part" JVM option, the simple command just hangs.... so MUST use the 3-part way described here: https://guides.dataverse.org/en/latest/developers/s3-direct-upload-api.html

OR better use dvuploader!!

Philip Durbin 🚀 (Jul 12 2024 at 13:13):

Interesting. It's especially interesting that the simple upload API works when direct upload is enabled (if the file is small enough).

Notification Bot (Jul 12 2024 at 19:46):

Sherry Lake has marked this topic as resolved.

Philip Durbin 🚀 (Sep 09 2024 at 18:01):

Related: Feature Request: Restrict API to require direct uploads #10828

Last updated: Oct 30 2025 at 06:21 UTC