jamie jamison (Sep 30 2024 at 17:28): jamie jamison (Sep 30 2024 at 17:28):We still have some separate s3 buckets for direct upload and original buckets. I'd like to get some of these merged to one bucket.
I didn't see bucket merging mentioned in documentation. Is there anything I should be planning or doing for this?
Thank you,
Jamie
jamie jamison (Sep 30 2024 at 18:15):I should probably explain. Earlier version of Dataverse required a separate s3 bucket for direct uploads. SInce that is no longer the case I'm trying to clean that up now.
Philip Durbin 🚀 (Sep 30 2024 at 18:26):Yeah, it makes sense. I just don't have experience with doing such a clean up. :sweat_smile:
jamie jamison (Sep 30 2024 at 18:34):I'm going to try with a couple of buckets on the test system but was hoping to hear about anyone else's experience.
jamie jamison (Sep 30 2024 at 18:46):@Don Sizemore Have you run across this issue?
Philip Durbin 🚀 (Sep 30 2024 at 19:17):maybe @Deirdre Kirmis has :shrug:
jamie jamison (Sep 30 2024 at 19:22):Hi @Deirdre Kirmis Have you ever run across this issue - merging s3 buckets.
Deirdre Kirmis (Sep 30 2024 at 19:59):We have always used only 1 bucket, but with separate 'stores' defined for direct upload. However, I think if you copy all of the folders/prefixes to one bucket, as long as you change the defined stores in your JVM options to point to the "new" bucket you should be okay. Or, if you just have the one store defined (ie: the merged store) .. you will just have to change the "storageidentifier" in the "dvobject" table to reflect the correct store that you have defined for each file. Does that make sense?
Deirdre Kirmis (Sep 30 2024 at 20:03):I think what i mean is, whatever is defined in the dvobject table as the storageidentier for the files should match the "store" that is defined in the domain.xml file. As long as that store points to the correct bucket it should work. Does that seem correct?
Philip Durbin 🚀 (Sep 30 2024 at 20:07):Sounds correct to me.
jamie jamison (Sep 30 2024 at 20:19):Yes, makes perfect sense. Thank you!
Don Sizemore (Oct 01 2024 at 12:40):@jamie jamison Deirdre is correct - you just want the datafiles where the storageidentifier field in the dvobject table points. there is some migration SQL in the bottom of Julian's "Helpful SQL queries" Google document that may help you do what you need.
Don Sizemore (Oct 01 2024 at 12:43):@jamie jamison see the "storage migration" queries at the bottom of https://docs.google.com/document/d/1-Y_iUduSxdDNeK1yiGUxe7t-Md7Fy965jp4o4m1XEoE/edit#heading=h.avuoo5kf0mdt - once you copy the datafiles into place where you want them, you would use queries like these to update the storageidentifier field?
jamie jamison (Oct 01 2024 at 18:08):Have to deal with production being down and then will get back to this. Thank you for all the info.
jamie jamison (Oct 03 2024 at 16:32):On the test system it looks like moving the buckets worked. Thank you all for the input.
Notification Bot (Oct 03 2024 at 16:33):jamie jamison has marked this topic as resolved.
Last updated: Oct 30 2025 at 06:21 UTC