Backwards compatibility of annotations

[profwillie]

I am having difficulty reading in annotations on 0.17.52.1 that were created in 0.15.49.1. It seems that there were changes to TimeIntervalAnnotation and TimeIntervalAnnotationSet that prevent the data from being deserialized. Is there a recommend workaround?

[danbohus]

Unfortunately, yes, the latest changes we have made to support multi-track annotations have led to a break in back-compatibility. One solution might be to use the old version to export the annotations to some other readable format that contains all the pertinent information (start and end times of annotations, values for each attribute, etc), and then have an app using the new psi version to read the exported file and generate a stream of TimeIntervalAnnotationSet objects that capture the annotations. If your annotations are single-track (they must be since you generated them with the old version), each new TimeIntervalAnnotationSet object will end up containing a single TimeIntervalAnnotation. You'll also need to generate a new annotation schema file (I think in old nomenclature this was the annotation definition file). If you are able to share the old schema, we can help explain what the new corresponding one might need to look like.

[iboucher125]

Thank you for your assistance. This is the old schema file:
LevelsOfNeedAnnotationDefinition.TXT

[danbohus]

So the sketch of the solution to migrate the annotations would look like this:

Step 1. Write a \psi application (using the old version of psi) that reads the annotation stream and writes it (perhaps via some Do operator) to a text file in an easily readable format.

Step 2. Create a new annotation schema file. Based on the schema you have sent above, I think the new annotation schema file should probably look something like this:

{
    "Name": "LevelsOfNeedAnnotationDefinition",
    "AttributeSchemas": [{
            "Name": "LevelsOfNeedAnnotation",
            "ValueSchema": {
                "$type": "Microsoft.Psi.Data.Annotations.EnumerableAnnotationValueSchema`1[[System.String, mscorlib]], Microsoft.Psi.Data",
                "DefaultValue": "0",
                "PossibleValues": [
		{
                 	"Value": "0"
                },
                {
                      "Value": "1",
                      "FillColor": "Red",
                      "TextColor": "White",
		},
		{
  		      "Value": "2",
                       "FillColor": "Green",
                       "TextColor": "White",
		},
		{
	 	      "Value": "3",
                      "FillColor": "Blue",
                      "TextColor": "White",
		},
		{
			"Value": "Flow",
                       "FillColor": "Yellow",
                       "TextColor": "Blue",
		},
                ]
            }
        }
    ]
}

Save this file under a name like LevelsOfNeed.schema.json

Step 3. Write a new \psi application (using the new version of psi) that reads all the information about your annotations from the text file you wrote at step 1, and creates a new stream with messages of type TimeIntervalAnnotationSet that will contain the new annotations. To do so, you will need to first load the new schema like so:

var annotationSchema = AnnotationSchema.LoadFrom(path + "LevelsOfNeed.schema.json");

To create a stream of TimeIntervalAnnotationSet objects corresponding to the annotations, you can either use some generator, like Generators.Sequence followed by a Select operator to create the corresponding TimeIntervalAnnotationSet, or you can try to leverage some of the existing ToTimeIntervalAnnotations(...) stream operators that exist in Microsoft.Psi.Data\Annotations\Operators.cs.

Either way, once you have the stream, you need to write it to a psi store, like this:

annotationsStream.Write(annotationSchema, $"Objects", outputStore);

Note that this is a bit different than the regular operator for writing psi streams to a store, in that the first parameter will be the annotationSchema object, then followed by the stream name and the output store.

Hope this helps, but if you go down this path and get stuck/have concrete questions, please continue posting to the issue.

[iboucher125]

We used a generator and wrote the stream to a new store. We can visualize the new annotation stream in PsiStudio and it appears to align with the audio stream. However, we are having difficulty reading from the store in a new \psi application. When comparing the metadata of the annotations generated from the CSV file and new annotations we created in PsiStudio using the same schema, it appears that the creation times are different. Could the creation time be a possible source of the problem and if so how can we change the creation time?

[danbohus]

To better understand the problem, could you provide a bit more details regarding:

1. what the values are for the creation and originating times of the messages on the streams, and the differences observed between them

2. how exactly you ran the pipeline (which parameters were passed to Pipeline.Run) and how the generator was written for writing the stream to the store

3. when you say you are having difficulty reading from the store in a new \psi application, in what way is that manifesting? Are you trying to read this stream together with other streams?