Stream: python

Topic: dvuploader support S3 - direct up load?


view this post on Zulip Sherry Lake (Sep 13 2024 at 15:27):

@Jan Range Does the python version support direct upload for us - AWS/S3 installations?

view this post on Zulip Philip Durbin ๐Ÿš€ (Sep 13 2024 at 15:29):

The README at https://github.com/gdcc/python-dvuploader says it does. Try it? :grinning:

view this post on Zulip Sherry Lake (Sep 13 2024 at 15:32):

Thanks... I was looking through the "yml" parameters.... expecting a setting there and didn't see it. I'll recommend it to my researchers.

view this post on Zulip Philip Durbin ๐Ÿš€ (Sep 13 2024 at 15:33):

Interesting. It seems like that config.yml file is optional.

view this post on Zulip Jan Range (Sep 13 2024 at 15:42):

@Sherry Lake yes, it is capable of performing the S3 upload :smile: By default it will attempt to use the Direct Upload API, but when it is not available at a collection will fall back to the Native API upload. That's why there is no specific option for the direct upload, but you can enforce the native upload.

Please be aware, there is currently a bug that throws a 500 Error but does not affect the upload. This is the corresponding issue and I will provide a fix soon.

view this post on Zulip Jan Range (Sep 13 2024 at 15:56):

Iโ€™m aiming to finalize the DVUploader app soon, which should make the upload process easier for your researchers. At the moment, Iโ€™m quite busy preparing for a workshop, but when this is over I'll have more time :muscle:

view this post on Zulip Philip Durbin ๐Ÿš€ (Sep 13 2024 at 16:16):

I can't wait for that app. :grinning:

view this post on Zulip Katie Mika (Mar 31 2025 at 18:40):

@Jan Range - regarding direct upload for the python dvuploader here, is it possible to include something similar to the -noIngest flag that the java dvuploader has?

view this post on Zulip Jan Range (Mar 31 2025 at 18:41):

@Katie Mika Yes, this is currently part of a PR, which will be merged soon.

https://github.com/gdcc/python-dvuploader/pull/24

Once merged, I will also update EasyDataverse to implement this feature and work with your other scripts :smile:

view this post on Zulip Katie Mika (Mar 31 2025 at 19:32):

Awesome, thank you!


Last updated: Nov 01 2025 at 14:11 UTC