César Ferreira (Dec 17 2025 at 15:46): César Ferreira (Dec 17 2025 at 15:46):Hi, I am trying to setup a trusted remote storage and I am encountering some errors. To give you a better context, I have an S3 bucket with Sentinel 2 data downloaded and my idea was to create dataset aggregating that data and use that bucket as a trusted remote storage. The settings I used to configure are as follows:
asadmin create-jvm-options '-Ddataverse.files.sentinel2.type=remote'
asadmin create-jvm-options '-Ddataverse.files.sentinel2.label=sentinel2'
asadmin create-jvm-options '-Ddataverse.files.sentinel2.base-store=s3'
asadmin create-jvm-options '-Ddataverse.files.sentinel2.base-url="https://stratus.d.incd.pt:8080/sentinel"'
asadmin create-jvm-options '-Ddataverse.files.sentinel2.download-redirect=true'
asadmin create-jvm-options '-Ddataverse.files.sentinel2.upload-redirect=true'
asadmin create-jvm-options '-Ddataverse.files.sentinel2.remote-store-name=Sentinel2'
asadmin create-jvm-options '-Ddataverse.files.sentinel2.upload-out-of-band=true'
I tried to create a file on the dataset with the following script:
export API_TOKEN=XXX
export SERVER_URL=https://repositorio.cloud.acnca.pt
export PERSISTENT_IDENTIFIER=doi:10.5072/FK2/PKUVLP
export JSON_DATA='{"description":"Test file","tabIngest": false,"categories":["Data"], "restrict":"false", "storageIdentifier":"sentinel2://DGT2015/S2A_MSIL1C_20150708T113056_N0500_R080_T29SMA_20231010T003232.SAFE/DATASTRIP/DS_S2RP_20231010T003232_S20150708T113621/MTD_DS.xml", "fileName":"MTD_DS.xml","mimeType":"application/xml","checksum": {"@type": "MD5", "@value": "e0cd675d26aa50e80f6a9d1992955db9"}}'
curl -X POST -H "X-Dataverse-key: $API_TOKEN" "$SERVER_URL/api/datasets/:persistentId/add?persistentId=$PERSISTENT_IDENTIFIER" -F "jsonData=$JSON_DATA"
but it failed and I got the following messages in server.log:
[2025-12-17T15:34:24.641+0000] [Payara 6.2025.3] [INFO] [] [] [tid: _ThreadID=128 _ThreadName=http-thread-pool::http-listener-1(16)] [timeMillis: 1765985664641] [levelValue: 800] [[
-------------------------------]]
[2025-12-17T15:34:24.642+0000] [Payara 6.2025.3] [INFO] [] [] [tid: _ThreadID=128 _ThreadName=http-thread-pool::http-listener-1(16)] [timeMillis: 1765985664642] [levelValue: 800] [[
jsonData: {"description":"Test file","tabIngest": false,"categories":["Data"], "restrict":"false", "storageIdentifier":"sentinel2://DGT2015/S2A_MSIL1C_20150708T113056_N0500_R080_T29SMA_20231010T003232.SAFE/DATASTRIP/DS_S2RP_20231010T003232_S20150708T113621/MTD_DS.xml", "fileName":"MTD_DS.xml","mimeType":"application/xml","checksum": {"@type": "MD5", "@value": "e0cd675d26aa50e80f6a9d1992955db9"}}]]
[2025-12-17T15:34:24.642+0000] [Payara 6.2025.3] [INFO] [] [] [tid: _ThreadID=128 _ThreadName=http-thread-pool::http-listener-1(16)] [timeMillis: 1765985664642] [levelValue: 800] [[
-------------------------------]]
[2025-12-17T15:34:24.880+0000] [Payara 6.2025.3] [WARNING] [] [edu.harvard.iq.dataverse.dataaccess.RemoteOverlayAccessIO] [tid: _ThreadID=128 _ThreadName=http-thread-pool::http-listener-1(16)] [timeMillis: 1765985664880] [levelValue: 900] [[
Not Supported: edu.harvard.iq.dataverse.dataaccess.RemoteOverlayAccessIO store with base store type: null]]
[2025-12-17T15:34:24.881+0000] [Payara 6.2025.3] [WARNING] [] [edu.harvard.iq.dataverse.ingest.IngestServiceBean] [tid: _ThreadID=128 _ThreadName=http-thread-pool::http-listener-1(16)] [timeMillis: 1765985664881] [levelValue: 900] [[
Failed to get file size, storage id, or failed to remove the temp tag on the saved S3 objectsentinel2://19b2cf2bef3-4aac2735d32e//DGT2015/S2A_MSIL1C_20150708T113056_N0500_R080_T29SMA_20231010T003232.SAFE/DATASTRIP/DS_S2RP_20231010T003232_S20150708T113621/MTD_DS.xml (Not supported)]]
[2025-12-17T15:34:24.881+0000] [Payara 6.2025.3] [WARNING] [] [edu.harvard.iq.dataverse.dataaccess.RemoteOverlayAccessIO] [tid: _ThreadID=128 _ThreadName=http-thread-pool::http-listener-1(16)] [timeMillis: 1765985664881] [levelValue: 900] [[
Not Supported: edu.harvard.iq.dataverse.dataaccess.RemoteOverlayAccessIO store with base store type: null]]
[2025-12-17T15:34:24.882+0000] [Payara 6.2025.3] [WARNING] [] [edu.harvard.iq.dataverse.ingest.IngestServiceBean] [tid: _ThreadID=128 _ThreadName=http-thread-pool::http-listener-1(16)] [timeMillis: 1765985664882] [levelValue: 900] [[
Error getting ingest limit for file: null : Not supported]]
[2025-12-17T15:34:24.882+0000] [Payara 6.2025.3] [INFO] [] [edu.harvard.iq.dataverse.ingest.IngestServiceBean] [tid: _ThreadID=128 _ThreadName=http-thread-pool::http-listener-1(16)] [timeMillis: 1765985664882] [levelValue: 800] [[
Incrementing recorded storage use by 0 bytes for dataset 4]]
Hmm. What version of Dataverse are you using? Also, can you please post to https://groups.google.com/g/dataverse-community as well so Jim sees it?
César Ferreira (Dec 17 2025 at 15:52):Version 6.8
Philip Durbin 🚀 (Dec 17 2025 at 16:37):Thanks for posting https://groups.google.com/g/dataverse-community/c/3faRy2l9iUc/m/z8gupidiCQAJ
Philip Durbin 🚀 (Dec 17 2025 at 19:28):Sounds like you got it working! Based on what I'm reading in that email thread.
César Ferreira (Dec 18 2025 at 10:47):Yes. The remote store had a base-store that didn't exist. After switching the value to the id of an existing storage driver I uploaded files from the remote storage successfully.
Philip Durbin 🚀 (Dec 18 2025 at 12:15):Great. Do you want to make a pull request to improve the documentation? Or is it good enough?
César Ferreira (Jan 09 2026 at 10:51):@Philip Durbin 🚀 As I shared in #community > Production examples of trusted remote store? I am currently trying to create a dataset with about 110k. That dataset has 1454 Sentinel 2 products, each one with about 50 files, and I have multiple datasets with more than 10k products that I also want to create in the future, probably those datasets would have about 500k files each.
Last updated: Jan 09 2026 at 14:18 UTC