Data Quality model extension

[hylkevds]

Since no one really knows what to do with the resultQuality DQ_Element property on Observation, we need something better.
Some proposals are currently being worked on.

[securedimensions]

One proposal / idea results from the work in the EU CitiObs project. It is understood that the use of the DQ_Element or the use of the Observation parameters puts heavy burdon on the database when filtering on Observations WHERE data-quality equals xyz.

One possible way to improve filtering performance is to introduce a seprate DataQuality entity that keeps the relevant information. Any observation meeting the detail expressed in a DataQuality instance is then simply linked. This allows to filter on the DataQuality entities and then fetch the associated observations.

The other aspect that was identified is that DataQuality of a sensor may depend on time. So, the extension proposal for the data model includes another entity HistoricalDataQuality. This allows to produce a timeseries for data quality.

The following figure illustrates the proposal. The details of the DataQuality entity and the naming used is potentially misleading. But, because we want to contribute to a standardized solution and stimulate a good discusion, we share the following diagram

The DataQuality entity is linked to Sensor allowing the expression of information that can be applied to obaservations created by the sensor. The DataQuality entity also links to Observation and the STAplus ObservationGroup which allows to separate data quality for single observations and a collection of observations. To express the timeseries of data quality related to a sensor, the HistoricalDataQuality is linke to Sensor.

The diagramm and all illustrations above represent the current snapshot of thinking. It will for sure evolve over the next month...

[humaidkidwai]

It's interesting to see DataQuality linked to Sensor, makes it easier to manage the maintenance of faulty Sensors.

Would it not be more appropriate to have the DataQuality linked to Datastream and store the DataQuality attribute in Datastream?

• one possible reason to do so would be to avoid redundancy, since all Observations in a Datastream are more likely to share certain attributes such as accuracy, precision, etc
• another reason could be the Quality of the data would be more viable to judge at an aggregate level rather individual Observations, some of which might be outliers.

The attributes for the DQ Entity do require more information

[schaefed]

The following proposal to represent quality information in STA 2.0 is focused on the requirements of dedicated tools for anomaly detection, like, for example, saqc.

Without going into too much detail here, the basic functionality of such tools can be described as follows: Software for anomaly detection can usually be understood as a collection of algorithms implemented as functions or methods, tailored towards a certain kind of anomaly (e.g., algorithms to detect outliers, scatter, constant values, ...). In the following, those functions/methods will be called 'quality measures'. Applied to a given data stream, those tools usually output some kind of quality information in the following called 'quality features', often also called "flags." When used, these tools are usually configured to apply certain quality measures on an input datastream. Those measures themselves are usually parameterized towards the characteristics of the input data. In most cases, several quality measures will be used to detect different kinds of anomalies within the same datastream. The output will be another datastream of quality features, usually, but not necessarily, of the same length as the input data. The exact representation of the quality features is heavily dependent on the use case, and to my knowledge, no widely accepted standards exist (although I'd love to be proven wrong on this). In practice, I have seen quite different quality feature schemes (i.e., definitions of quality features and their interrelation), ranging from a set of concrete labels (like "OK," "SUSPICIOUS," "BAD") over integers within given bounds (1-10, 0-255) to 'continuous' representation as real numbers like in the interval [0, 1]. The main point here is that basically everything could be a quality feature and therefore part of a quality feature scheme.

The representation of quality features, especially if this representation should be traceable or even reproducible, needs to fulfill several requirements:

• It should be flexible enough to allow quality features of any data type.
• It needs a way to provide meta information to the quality measure (i.e., what kind of algorithm was applied, a reference to the method description and/or implementation).
• It needs to store the parameters used to adjust a quality measure to a given data stream.
• It should have the possibility to 'bundle' several quality measures into a larger setup.

The following diagram shows a possible realization of those requirements as an extension of the STA 1.1 data model:

This scheme allows describing different QualityMeasure, to bundle them together with concrete arguments into a QualityScheme, which is related to the actual QualityFeatures. A QualityFeature is either associated with an Observation or a Datastream (to also provide aggregated quality information). The QualityFeature itself is specified by the QualityScheme, which at least defines its data type.

[securedimensions]

First approach of a consolidated Data Model

STA version 1.1

STAplus version 1.1

[schaefed]

The latest version of the consolidated model:

[hylkevds]

For V2 we're renaming the parameters field to properties, so that should probably also be used here.
There seems to be a Typo in the diagram: HistoricalQualitylFeature

[hylkevds]

Question about the cardinalities:
HistoricalQualityFeature can have multiple Sensors and multiple Observations. What does this mean?
I would expect a HistoricalQualityFeature to be the QualityFeature(s) a single Sensor or Observation had at a given point in time, so I would expect the HistoricalQualityFeature to have 1 Sensor or 1 Observation.

Also, how many QualityFeatures can an Observation have? The latest diagram has 0..1 but the original had 0..*.

[hylkevds]

We may get questions about the term QualityFeature, and its overlap with the existing Feature class. Are there alternative names that could be considered?

[nbrinckm]

Regarding the term: How about „QualityAnnotation“?