Marcos Anjos (Nov 06 2025 at 15:50): Marcos Anjos (Nov 06 2025 at 15:50):Good afternoon,
I'm currently working on a task to migrate our Dataverse installation from a VM-based setup to a new Docker-based architecture. I have a question regarding the migration of production data (version 5.12.1), including users, metadata, and files, to the new 6.8 version.
Is there a recommended strategy for backing up and migrating all data between these versions?
I’ve checked the official documentation, which mentions the option of migrating data between repositories, but I’d like to confirm whether that is the only available approach or if there are better methods for this kind of upgrade.
My initial plan is:
Could you advise on the best approach to ensure data integrity and consistency throughout the migration process?
Philip Durbin 🚀 (Nov 06 2025 at 17:37):@Marcos Anjos hi! Unfortunately, you just missed our Containerization Working Group meeting. This would have been a great topic? Would you like us to put it on the agenda for next month? Here's our website: https://ct.gdcc.io
Philip Durbin 🚀 (Nov 06 2025 at 17:38):Meanwhile, we'll try to get you some answers but I'll probably need to lean on others' expertise.
Philip Durbin 🚀 (Nov 06 2025 at 17:38):Generally, I'll say that your plan sound fine! Please let us know if find any bumps in the road!
Oliver Bertuch (Nov 06 2025 at 23:18):Philip Durbin 🚀 said:
Meanwhile, we'll try to get you some answers but I'll probably need to lean on others' expertise.
Just tag @Don Sizemore and me :stuck_out_tongue_closed_eyes: :wink:
Oliver Bertuch (Nov 06 2025 at 23:19):Welcome @Marcos Anjos and congrats on choosing containers! You're in for a fun joyride :smiling_imp: But don't worry, this is #troubleshooting and we're here to help.
Oliver Bertuch (Nov 06 2025 at 23:20):For the migration, let me highlight some bits I'm currently doing with my own container to container migration from 4.20 to 6.8...
Oliver Bertuch (Nov 06 2025 at 23:21):It's really handy to have a snapshot of the database around for experiments. With containers, you can do most of these experiments on your laptop. For many scenarios it's not necessary to do this on some beefy machine.
Oliver Bertuch (Nov 06 2025 at 23:24):In case you don't want to jump through the releases one by one, which is the only officially supported way of doing upgrades, there's a few ways you can go about this.
Oliver Bertuch (Nov 06 2025 at 23:24):Some have done dumps of the metadata and re-importing in a fresh installation.
Oliver Bertuch (Nov 06 2025 at 23:24):I will not do it like that. Here's what I presented at DCM2025 about my upgrade adventure: http://talks.bertuch.name/dcm2025-upgrade/#/
Oliver Bertuch (Nov 06 2025 at 23:27):The most important bit around the actual data is that you need to be careful how you move it around.
Oliver Bertuch (Nov 06 2025 at 23:28):Depending on the storage system you're using, you need to make sure not to alter the "identifier" of the storage system in the config. Otherwise, you will have to do a data migration in the database to make it available again (the ID is part of the information stored about a file in the database)
Oliver Bertuch (Nov 06 2025 at 23:29):Also, if you plan on moving the data between places, you must make sure the structure is all the same.
Oliver Bertuch (Nov 06 2025 at 23:30):If you are planning to migrate between storage types, things will become a bit more tricky... :see_no_evil: You'll probably be best of with a data migration for it, so you can have the new config in the upgraded instance
Oliver Bertuch (Nov 06 2025 at 23:31):For your experiments you should know that in theory, it is enough to have Dataverse and Postgres running.
Oliver Bertuch (Nov 06 2025 at 23:31):This will at least let you make sure the database is still available. But using a browser, the UI will not show you anything because Solr will be either missing or empty.
Oliver Bertuch (Nov 06 2025 at 23:32):While it is possible to migrate Solr data, I'd advise against doing it. The reason is simple: 6.0 will require a full re-index anyway. So why bother moving that data around just to throw it away.
Oliver Bertuch (Nov 06 2025 at 23:34):Of course, if you are using fulltext indexing in your instance, you will need to either disable it during experiments or make sure that you have a connection to your data. Just a word of caution: this will take a long time depending on your data amount...
Oliver Bertuch (Nov 06 2025 at 23:34):If you have custom metadata schemas, don't forget to apply the necessary changes to the latest schema.xml. Otherwise your re-index is probably going to fail, as Dataverse will not be able to add the metadata from the DB to the Index.
Oliver Bertuch (Nov 06 2025 at 23:36):That's all I have for now!
Mahendra Paipuri (Jun 03 2026 at 12:39):Hello @Oliver Bertuch We would like to do something similar, albeit, upgrading from 5.14 to latest 6.10.1. We have the current instance installed on a VM "manually" and we will migrate it to a different VM on Rocky 9 for the newer version. The directory hierarchy will be exactly same and we will move the actual files from older VM to newer VM at the same exact path.
My plan is to:
It worked although I am not quite sure if entire DB schema is up to date. So, what I did is to install a "fresh" Dataverse instance with version 6.10.1 and compared the DB schema between the fresh instance and migrated instance (from 5.14 to 6.10.1). There are few differences in the schema and I applied those migrations as well on our migrated instance and now I have exactly the same DB schema on a fresh instance and migrated instance.
Can we be sure that our migrated instance is clean enough and we will not have any future issues?
Cheers!
Oliver Bertuch (Jun 03 2026 at 12:45):Sorry @Mahendra Paipuri that's impossible for me to say. There are too many variables here and things that may have gone wrong or not. In theory you should be fine. But I can't and won't give any guarantees for that.
Oliver Bertuch (Jun 03 2026 at 12:46):My own experimental approach to this can be found in the Dataverse Recipes. Again: no guarantees.
Mahendra Paipuri (Jun 03 2026 at 12:49):Alright, I get it!! Cheers @Oliver Bertuch
Mahendra Paipuri (Jun 03 2026 at 12:52):I have a question out of curiosity. So, when I looked at the differences in DB schemas between fresh instance and migrated instance, one of the statement is to drop table doidataciteregistercache. When I grepped on Dataverse repo (https://github.com/IQSS/dataverse/tree/develop/src/main/resources/db/migration), I did not find any references to this table being created and dropped. So, I was bit lost on how DB migrations are managed in Dataverse. Would you happend to have any insights on this @Oliver Bertuch
Oliver Bertuch (Jun 03 2026 at 12:53):https://guides.dataverse.org/en/latest/developers/sql-upgrade-scripts.html
Mahendra Paipuri (Jun 03 2026 at 12:59):That is very helpful!! CHeers again @Oliver Bertuch
Philip Durbin 🚀 (Jun 03 2026 at 13:09):@Mahendra Paipuri good luck with your upgrade!
Last updated: Aug 18 2026 at 08:27 UTC