Stream: community

Topic: api curl upload utf-8 filenames

Péter Pallinger (Jun 11 2026 at 15:21):

I am trying to upload files with accented filenames.
Example try:

curl -f -H "X-Dataverse-key:$API_KEY" -X POST -F "file=@\"Boldogság\"" -F '
jsonData={"filename": "Boldogs%C3%A1g"}' "$SERVER/api/datasets/:persistentId/add?persistentId=$PID"

This results in "Boldogság" as filename. If I leave the jsonData without urlencoding, the result is the same. If I use utf-8 escape sequences (like \045F), the result is the same.
I can easily rename the file on the user interface, but I would not like to do that to hundreds of files...

Philip Durbin 🚀 (Jun 11 2026 at 15:24):

Interesting. You definitely want to use curl? I'm just wondering if pyDataverse handles this.

Péter Pallinger (Jun 12 2026 at 04:41):

I will try pydataverse. Unfortunately, uploading big files with pydataverse is really slow and memory hungry, or - after 2GB - even impossible.

Philip Durbin 🚀 (Jun 12 2026 at 09:50):

Maybe you could try DVUploader as well: https://guides.dataverse.org/en/6.10.1/user/dataset-management.html#command-line-dvuploader

Philip Durbin 🚀 (Jun 12 2026 at 09:50):

To be clear, I'm wondering if there's a bug in the backend (Dataverse) or if we need add a flag or something to curl.

Péter Pallinger (Jun 12 2026 at 12:22):

I triesd uploading with pydataverse, and the error is the same: all non-8bit UTF-8 characters are represented with two characters -- the two bytes for the actual UTF-8 characters interpreted in a different code page (iso-8859-1 maybe?). If this is not a bug in the backend, it seems at least an outdated default codepage for processing input.
How can the uploaded content's encoding be specified?

Péter Pallinger (Jun 12 2026 at 12:37):

I also tried dvuploader, but it failed to upload the file with a 400 error (it connected good, and detected if the file MD5 existed before, and nothing was logged in the server logs). So it failed before the two other methods.

Philip Durbin 🚀 (Jun 12 2026 at 13:21):

Boo. I'm calling for reinforcements. :sweat_smile:

Philip Durbin 🚀 (Jun 12 2026 at 13:22):

@Péter Pallinger one more thing to try if you're willing! https://github.com/gdcc/python-dvuploader

Péter Pallinger (Jun 12 2026 at 14:56):

python-dvuploader did upload the file, but the resulting file name was the same...

Philip Durbin 🚀 (Jun 12 2026 at 15:23):

bah

Péter Pallinger (Jun 17 2026 at 12:13):

I filed a bug report:
https://github.com/IQSS/dataverse/issues/12470

There is also a working workaround: You can rename the file after upload. But this is quite slow in case of many small files.

Philip Durbin 🚀 (Jun 17 2026 at 13:08):

@Péter Pallinger thanks!

Last updated: Aug 18 2026 at 08:27 UTC