Stream: containers

Topic: Region issue for dataverse using AWS S3 direct upload

view this post on Zulip Guillermo Portas (May 28 2024 at 15:56):

I want to run a containerized Dataverse instance using a direct upload s3 storage. I want to run it pointing to a real AWS account and bucket, that is, without localstack and minio.

I have used the microprofile config options for setting the AWS credentials (files.<id>.access-key and files.<id>.secret-key) and added them in the JVM_ARGS Dataverse env variable.

services:
  test_dataverse:
    container_name: 'test_dataverse'
    hostname: dataverse
    image: ${DATAVERSE_IMAGE_REGISTRY}/gdcc/dataverse:${DATAVERSE_IMAGE_TAG}
    restart: on-failure
    user: payara
    environment:
      DATAVERSE_DB_HOST: postgres
      DATAVERSE_DB_PASSWORD: secret
      DATAVERSE_DB_USER: ${DATAVERSE_DB_USER}
      JVM_ARGS: -Ddataverse.files.storage-driver-id=file1
        -Ddataverse.pid.providers=fake
        -Ddataverse.pid.default-provider=fake
        -Ddataverse.pid.fake.type=FAKE
        -Ddataverse.pid.fake.label=FakeDOIProvider
        -Ddataverse.pid.fake.authority=10.5072
        -Ddataverse.pid.fake.shoulder=FK2/
        -Ddataverse.files.s3.type=s3
        -Ddataverse.files.s3.label=XXX
        -Ddataverse.files.s3.bucket-name=XXX
        -Ddataverse.files.s3.upload-redirect=true
        -Ddataverse.files.s3.download-redirect=true
        -Ddataverse.files.s3.ingestsizelimit=50000000
        -Ddataverse.files.s3.url-expiration-minutes=60
        -Ddataverse.files.s3.connection-pool-size=2048
        -Ddataverse.files.s3.access-key=XXX
        -Ddataverse.files.s3.secret-key=XXX
        # No mpconfig for region?
    ports:
      - '8080:8080'
    networks:
      - dataverse
    depends_on:
      - test_postgres
      - test_solr
    tmpfs:
      - /dumps:mode=770,size=2052M,uid=1000,gid=1000
      - /tmp:mode=770,size=2052M,uid=1000,gid=1000
    mem_limit: 2147483648 # 2 GiB
    mem_reservation: 1024m
    privileged: false

When I run the containers and Dataverse tries to access S3  an SdkClientException is thrown, describing that it could not find the region
Screenshot-2024-05-28-at-16.34.22.png
How can I specify the region?

view this post on Zulip Oliver Bertuch (Jun 07 2024 at 07:02):

Hi @Guillermo Portas. Please note that the storage subsystem has not yet been completely MPCONFIG enabled. The only options enabled to use MPCONFIG are the keys.

view this post on Zulip Oliver Bertuch (Jun 07 2024 at 07:03):

It seems like the option to configure a "real" region is actually missing from the S3 configuration options!

view this post on Zulip Oliver Bertuch (Jun 07 2024 at 07:04):

You can define a custom-region though. Let me check what's in the code for this and if you can use it to override things.

view this post on Zulip Oliver Bertuch (Jun 07 2024 at 07:05):

Otherwise, the only other option to configure these regions I'm aware of would be a ~/.aws/config entry

view this post on Zulip Oliver Bertuch (Jun 07 2024 at 07:08):

Nope - the custom endpoint region thing is only available when using a custom endpoint URL

view this post on Zulip Oliver Bertuch (Jun 07 2024 at 07:08):

You could try to define the official endpoint URL as custom URL :smile_cat:

view this post on Zulip Philip Durbin 🚀 (Jun 07 2024 at 11:44):

Yes, we talked about this on May 30th at a container meeting: https://harvard.zoom.us/rec/share/FVYyxSpMkOLX-jo9TLCBek0KhkZiFZ6e4v4Nka7M4eSpbVczSxrn5vLcKHG9if9x.cBqXOiQlyKpDN8pD

view this post on Zulip Philip Durbin 🚀 (Jun 07 2024 at 11:44):

As Oliver is saying, one has to use custom-region for now.

view this post on Zulip Guillermo Portas (Jun 10 2024 at 12:54):

Thank you @Oliver Bertuch and @Philip Durbin !

view this post on Zulip Guillermo Portas (Jun 10 2024 at 12:55):

I finally decided to use localstack for my testing setup, but I will take your comments into account for the future.

Last updated: Oct 30 2025 at 05:14 UTC