downloading all files from dataset with private URL · community

Stream: community

Topic: downloading all files from dataset with private URL

Péter Pallinger (Jan 29 2026 at 14:05):

If a reviewer gets a private URL to an unpublished dataset, they may want to download multiple files from it, but the only way to do that currently seems to be through the web interface. However, the zipped download does not work for a larger total file size.
API download does not work. Even though you _can_ get the file list for the unpublished dataset through the API, downloading the files is forbidden (it seems to me, that it requires a JSESSIONID cookie).
Is there a true and tried way to download large datasets using private/preview URLs?

Philip Durbin 🚀 (Jan 29 2026 at 16:15):

You should be able to use the preview URL token as an API token. Does that help?

Dieuwertje Bloemen (Jan 29 2026 at 16:15):

It's quite a coincidence, because this week we finally have a production ready Globus integration that can also do exactly this :sweat_smile: . My colleague Eryk is working on making his work available on GitHub. Hopefully tomorrow or next week we can share it. We worked on improving Globus so we can use it for restricted access data and with private urls. The only thing that isn't working is anonymized private urls (the ones where the author name etc. gets redacted). I'll drop the url to the GitHub once I have it.

Péter Pallinger (Jan 30 2026 at 11:09):

Philip Durbin 🚀 said:

You should be able to use the preview URL token as an API token. Does that help?

That is a good tip, thank you. I did not see it in the documentation... I already implemented a workaround by getting a JSESSIONID, but this would be cleaner.