Stream: troubleshooting

Topic: ✔ DVUploader with Large file

Sherry Lake (Mar 21 2024 at 17:01):

Any advice on timeouts I am getting with DVUploader - uploading 350GB file. I am getting timeout errors, I know why, just need to figure out how to keep the S3 "URLs" - parts open and not time out.

Error response: HTTP/1.1 403 Forbidden : <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Request has expired</Message><X-Amz-Expires>3599</X-Amz-Expires><Expires>2024-03-21T16:18:09Z</Expires><ServerTime>2024-03-21T16:18:14Z</ServerTime><RequestId>4FH0FYK8QHER1BZP</RequestId><HostId>Cnn6zpUyPNgTnWd7lgMUa3WTnO62P7Y3v0uORF+NszVmh2FsbD0ceztzUDYQhgelZ5J7Tp60gDg=</HostId></Error>
Retrying upload of part: 118
Upload of part: 118 failed with status: 403 (retrying)

Might be a Jim question, which I asked in Slack.

@Deirdre Kirmis How did you get over the "time out", or are you using S3?
Thanks.

Sherry Lake (Mar 21 2024 at 17:08):

Update from Jim in Slack:

Yes - there's a dataverse.files.<id>.url-expiration-minutes setting described in https://guides.dataverse.org/en/latest/installation/config.html#second-configure-your-dataverse-installation-to-use-s3-storage. The default is 60 minutes.

Deirdre Kirmis (Mar 21 2024 at 17:20):

looking into this to remember .. i do know that we had to set Timeout and ProxyTimeout in our apache config as well as the database timeout option

Deirdre Kirmis (Mar 21 2024 at 17:21):

Timeout 6000
ProxyTimeout 6000

Deirdre Kirmis (Mar 21 2024 at 17:23):

and we are using S3

Deirdre Kirmis (Mar 21 2024 at 17:29):

here is the command that worked for us at the time, but I think there have been changes (ie: not requiring the -directupload flag)

java -jar DVUploader-v1.2.0beta3.jar -recurse -directupload -key=<your api token> -did=doi:10.48349/ASU/<your dataset code> -server=https://dataverse.asu.edu <directory path>

Deirdre Kirmis (Mar 21 2024 at 22:06):

do you have CORS rules set on your s3 bucket?

Sherry Lake (Mar 25 2024 at 14:28):

Here are our CORS rules:

[
    {
        "AllowedHeaders": [
            "*"
        ],
        "AllowedMethods": [
            "GET",
            "PUT"
        ],
        "AllowedOrigins": [
            "*"
        ],
        "ExposeHeaders": [
            "ETag"
        ],
        "MaxAgeSeconds": 0
    }
]

Deirdre Kirmis (Mar 25 2024 at 15:31):

that looks okay .. i think .. i finally found the email thread that I had with Jim M when we were trying to make this all work, and I was getting the exact same error ..

.. i need to compile this whole thing into some kind of guide or something because there is so much valuable info that we got from jim's brilliant mind in it about how direct upload works .. i will send it to you

.. one thing that I can see is that we set the dataverse.files.<id>.url-expiration-minutes to 480 .. what is yours set to? and we definitely used the 1.1.2 beta version of dvuploader .. an yea the proxy timeouts didn't really matter

Oliver Bertuch (Mar 25 2024 at 15:39):

Adding more docs, especially for troubleshooting, to the guide would be much appreciated!

Deirdre Kirmis (Mar 25 2024 at 15:53):

okay, I will see if I can organize it a little and then send it on :smile:

Sherry Lake (Mar 28 2024 at 19:44):

setting this parameter worked

:there's a dataverse.files.<id>.url-expiration-minutes setting described in https://guides.dataverse.org/en/latest/installation/config.html#second-configure-your-dataverse-installation-to-use-s3-storage. The default is 60 minutes.

Sherry Lake (Mar 28 2024 at 19:49):

I tried to mark this as resolved, but could not resolve - "issues" older than 7 days. So if someone with more power than me can resolve, Thanks.

Notification Bot (Mar 28 2024 at 19:51):

Philip Durbin has marked this topic as resolved.

Philip Durbin 🚀 (Mar 28 2024 at 19:53):

Done! I resolved this topic.

Last updated: Oct 30 2025 at 06:21 UTC