Stream: community

Topic: CustomMetadata Migration

Johannes D (Sep 15 2023 at 14:34):

Metadata blocks can be updated, the technical process is similar to the initial integration. However, it is not yet documented how changes affect already captured metadata or the existing structure. The current implementation [1] simply overwrites the existing definitions, matching is based on the value of the "name" property of metadata blocks and dataset fields (or "value" for elements of the controlled vocabulary). As a result, names are globally unique and not per metadata block. Therefore, the behavior when updating metadata blocks must be considered undefined and risky!

To improve the situation it would be great if we could document and answer the following questions:

• What processes must be followed to update a metadata block?
  • In which cases can I just follow the existing process without any side effects?
    • Example: changing the "description" or "displayOrder" property? Adding new properties?
• In what cases are additional steps required?
  • Example: How to delete a property?
• In which cases is it necessary to migrate the metadata?
  • How can I perform the data migration?
    [1] https://github.com/IQSS/dataverse/blob/develop/src/main/java/edu/harvard/iq/dataverse/api/DatasetFieldServiceApi.java

Philip Durbin 🚀 (Sep 18 2023 at 12:19):

I absolutely agree. It should definitely be documented. @Johannes D would you be able to create an issue about this?

Henning Timm (Sep 18 2023 at 12:22):

I faced this problem recently. I would be interested in this as well. I can at least add a user story or two ;)

Philip Durbin 🚀 (Sep 18 2023 at 12:27):

That would be much appreciated!

Johannes D (Sep 19 2023 at 06:16):

Sure, can I'm also happy to write the documentation but I need some help grasping the functionality.

Johannes D (Sep 19 2023 at 09:41):

@Philip Durbin : I'm trying to understand the source code related to metadatablocks. I struggle to understand the purpose of DatasetFieldDefaultValue. My editor says there is no usage of the class/object. Is that correct?

Philip Durbin 🚀 (Sep 19 2023 at 10:37):

I'm seeing 14 usages:

Screenshot-2023-09-19-at-6.37.01-AM.png

Philip Durbin 🚀 (Sep 19 2023 at 10:46):

I do find it interesting that the project still compiles if I delete DatasetFieldDefaultValue and DefaultValueSet and only these lines:

diff --git a/src/main/java/edu/harvard/iq/dataverse/DatasetFieldType.java b/src/main/java/edu/harvard/iq/dataverse/DatasetFieldType.java
index 824b486a42..b328d46f3a 100644
--- a/src/main/java/edu/harvard/iq/dataverse/DatasetFieldType.java
+++ b/src/main/java/edu/harvard/iq/dataverse/DatasetFieldType.java
@@ -504,17 +504,6 @@ public class DatasetFieldType implements Serializable, Comparable<DatasetFieldTy
         this.datasetFields = datasetFieldValues;
     }

-    @OneToMany(mappedBy = "datasetField", cascade = {CascadeType.REMOVE, CascadeType.MERGE, CascadeType.PERSIST})
-    private List<DatasetFieldDefaultValue> datasetFieldDefaultValues;
-
-    public List<DatasetFieldDefaultValue> getDatasetFieldDefaultValues() {
-        return datasetFieldDefaultValues;
-    }
-
-    public void setDatasetFieldDefaultValues(List<DatasetFieldDefaultValue> datasetFieldDefaultValues) {
-        this.datasetFieldDefaultValues = datasetFieldDefaultValues;
-    }
-
     @Override
     public int compareTo(DatasetFieldType o) {
         return Integer.compare(this.getDisplayOrder(), (o.getDisplayOrder()));

Philip Durbin 🚀 (Sep 19 2023 at 10:47):

Maybe it was an experiment? And isn't actually used? I'm not sure.

Johannes D (Sep 19 2023 at 10:58):

I ignored the self usage. I tried the same and stated a new instance, toyed around and found no issues. Do you think it is save to remove the class?

Philip Durbin 🚀 (Sep 19 2023 at 11:05):

I'd want to ask around internally first.

Oliver Bertuch (Sep 19 2023 at 11:34):

On a related note, here is my list of shortcomings I found in the metadata block spec:

1. Headers all required or not?
2. Headers may not contain other whitespace
3. Tabs of non-required, empty values required or not?
4. CV header line must be present even if no CVs
5. Do not allow fields from other block in list. (Spec says it's not good practice only)
6. blockUri required or not?
7. parent field must be of type "none" (only mentioned in type defs)
8. displayOrder for subfields not clearly defined
9. allowed nesting levels of fields not clearly defined
10. how to make comments
11. what about empty / blank lines
12. what about order of headings
13. Can a compound field parent be facetable? (Probably not because format unclear)

Johannes D (Sep 19 2023 at 11:35):

@Philip Durbin I believe the same applies to DatasetDistributor as there is no object created in the codebase.. and DatasetKeyword without any usage.

Philip Durbin 🚀 (Sep 19 2023 at 11:49):

@Johannes D maybe you could make a PR to delete all of it. I assume tests will pass. We can point other devs at it and ask if we need it.

Johannes D (Sep 19 2023 at 13:12):

Philip Durbin said:

Johannes D maybe you could make a PR to delete all of it. I assume tests will pass. We can point other devs at it and ask if we need it.

here it is: https://github.com/IQSS/dataverse/pull/9932

Philip Durbin 🚀 (Sep 19 2023 at 13:14):

@Leo Andreev what do you think?

Last updated: Nov 01 2025 at 14:11 UTC