Mattias de Hollander (Mar 03 2025 at 15:49): Mattias de Hollander (Mar 03 2025 at 15:49):I am trying out python-dvuploader for uploading large datasets. Is there a public instance available for testing that support direct uploads? The ones I have tested fall back to the native api. @Jan Range , do you have suggestions?
Jan Range (Mar 03 2025 at 15:54):@Mattias de Hollander thanks for reaching out! Unfortunately, there is no public direct upload Dataverse instance to my knowledge. If it's solely for testing purposes, I recommend using Docker to host it locally. Would this be an option for you? Happy to help setting it up :smile:
Mattias de Hollander (Mar 03 2025 at 16:06):Ok, than I don't look any further. And thanks for the fast reply and developing this client/library. Running it locally with docker is possible for sure. I give that a try.
Jan Range (Mar 03 2025 at 16:07):Thanks a lot! Let me know if you need any help :smile:
Philip Durbin 🚀 (Mar 03 2025 at 17:09):@Mattias de Hollander the default Dataverse dev environment includes LocalStack with direct upload enabled: https://guides.dataverse.org/en/6.5/developers/dev-environment.html#quickstart
This line: https://github.com/IQSS/dataverse/blob/v6.5/docker-compose-dev.yml#L40
Jan Range (Mar 03 2025 at 17:14):Also a note, when using LocalStack, set the DVUPLOADER_TESTING environment variable to True. Otherwise, the received URLs internally won't work, since these are aliased by Docker. The URL http://localstack/... will then be converted to http://localhost/....
Jan Range (Mar 03 2025 at 17:14):https://github.com/gdcc/python-dvuploader?tab=readme-ov-file#development
Philip Durbin 🚀 (Mar 03 2025 at 17:23):@Mattias de Hollander so, I just learned that direct upload is enabled on https://beta.dataverse.org (on the default store) so you could try your test there.
Mattias de Hollander (Mar 04 2025 at 09:44):Good to hear direct upload is enabled on the beta dataverse website. Has been a while since I worked on dataverse and was not aware of this instance. However, I can register, but I think I need an elevated role to be able to add data, or create a new dataverse first. Can someone do that? Thanks.
In the mean time I give spinning up a docker instance a try.
Philip Durbin 🚀 (Mar 04 2025 at 13:22):Are you able to create a dataverse (we're starting to call these "collections" by the way).
Mattias de Hollander (Mar 04 2025 at 13:40):Not at beta.dataverse.org. Good naming update :)
Philip Durbin 🚀 (Mar 04 2025 at 14:11):@Mattias de Hollander it says, "Anyone with a Dataverse account can add sub dataverses":
Screenshot 2025-03-04 at 9.10.55 AM.png
Philip Durbin 🚀 (Mar 04 2025 at 14:12):You don't see an "Add Data" button?
Mattias de Hollander (Mar 04 2025 at 14:26):I do see that button. Here is my view:
image.png
And the response for both options is this:
image.png
Philip Durbin 🚀 (Mar 04 2025 at 14:27):500 error?!? D'oh! 
Philip Durbin 🚀 (Mar 04 2025 at 14:28):org.postgresql.util.PSQLException: ERROR: column "displayoncreate" does not exist
Philip Durbin 🚀 (Mar 04 2025 at 14:29):Shoot. We're working a fix for this. Please disregard what I said. Please don't use the beta server until this is fixed. I'll let you know!
Mattias de Hollander (Mar 04 2025 at 14:30):No problem. Happy when it is there, otherwise local options take more time but also work out. Thanks anyways.
Philip Durbin 🚀 (Mar 04 2025 at 14:37):If you create a collection on https://demo.dataverse.org I can point it to store with S3 direct upload configured. Lemme know.
Mattias de Hollander (Mar 04 2025 at 15:37):That would be great. Here is one: https://demo.dataverse.org/dataverse/mhtest1
Philip Durbin 🚀 (Mar 04 2025 at 16:02):Ok, please try it now.
Mattias de Hollander (Mar 04 2025 at 16:26):Thanks! I give it a try tomorrow and let you know.
Mattias de Hollander (Mar 05 2025 at 08:26):Super, it works now with direct upload support. I will start exploring it. Thanks a lot for the support.
Philip Durbin 🚀 (Mar 05 2025 at 12:09):Yep, sure.
Mattias de Hollander (Mar 11 2025 at 14:31):I've started testing the S3 direct upload support, thanks for making this possible. If you have any ideas regarding my follow-up question about setting parameters for direct upload with python-dvuploader (https://dataverse.zulipchat.com/#narrow/channel/432390-large-data/topic/setting.20parameters.20for.20direct.20upload.20with.20python-dvuploader/with/504607881), I'd appreciate your input.
Last updated: Oct 30 2025 at 06:21 UTC