Stream: ui-dev

Topic: File storage and upload on SPA

view this post on Zulip Paul Boon (Aug 06 2025 at 09:19):

The developer documentation suggest that you need S3 credentials to be able to upload a file. And indeed, when trying to upload it fails with : FileJSDataverseRepository.ts:299 PUT https://beta-dataverse-direct.s3.us-east-1.amazonaws.com/10.5072/FK2/HWDGFW…Signature=286e552… 403 (Forbidden).
It seems that the amazonaws is hardwired, which does not work wel for non-IQSS developers.
It would be better if the develop enviromment used a minio docker container.

But futhermore, it might be that the Dataverse backend also must use this same S3 as its file storage for the upload to work?

view this post on Zulip Philip Durbin 🚀 (Aug 06 2025 at 13:06):

I'm confused. In development we do use MinIO and LocalStack. In this test, for example: https://github.com/IQSS/dataverse/blob/v6.7.1/src/test/java/edu/harvard/iq/dataverse/api/S3AccessIT.java

So there shouldn't be any dependence on AWS. Do we need to fix the developer docs? :thinking:

view this post on Zulip Paul Boon (Aug 06 2025 at 13:09):

I am looking at the docs: https://github.com/IQSS/dataverse-frontend/blob/develop/DEVELOPER_GUIDE.md#running-the-project-locally
If I run the SPA locally, I get that error.
Sorry for the confusion, I added 'on SPA' to the topic.

view this post on Zulip Philip Durbin 🚀 (Aug 06 2025 at 13:16):

thanks

view this post on Zulip Philip Durbin 🚀 (Aug 06 2025 at 13:19):

I see what you mean:

# /dev-env/ directory

# copy the .env.example file to .env
# To test file upload, update the .env file with S3 credentials
$ cp .env.example .env

Yes, my .env.file has these:

S3_ACCESS_KEY=REDACTED
S3_SECRET_KEY=REDACTED

I agree this is suboptimal. I assumed it was using MinIO or LocalStack rather than actual AWS S3. :thinking:

view this post on Zulip Philip Durbin 🚀 (Aug 06 2025 at 13:21):

Ha, I see from my Slack history that @Guillermo Portas shared those secrets with me about a year ago.

@Guillermo Portas I forget the details on this. Is the frontend dev environment dependent on AWS S3? Can developers use MinIO or LocalStack instead?

view this post on Zulip Guillermo Portas (Aug 06 2025 at 15:17):

The original idea was to use MinIO / LocalStack, but we ultimately discarded that approach due to difficulties getting it to work with direct uploads.
To speed up development, we decided to switch to a real S3 bucket instead.
We're happy to transition to a fully containerized solution using MinIO / LocalStack in the future, as long as we can get a working setup.

view this post on Zulip Philip Durbin 🚀 (Aug 06 2025 at 15:22):

Great, I added this to our "Roadmap - Technical Brainstorming" doc.

Last updated: Nov 01 2025 at 14:11 UTC