STAC Data Keys update

[mike-gangl]

STAC Data Keys Updates

Currently we use STAC asset keys of "data", "metadata" and others. This disallows the usage of multiple "data" types as an asset, which breaks the SBG workflows that contain multiple data workflows.

Acceptance Criteria

• STAC asset key is not based on "data" or "metadata" - should be unique and not known ahead of time and based on filename

• STAC asset role is used to define data, metadata, etc. (https://github.com/radiantearth/stac-spec/blob/master/item-spec/item-spec.md#asset-roles )

• Ingest commands can ingest multiple "data" role types successfully

• stage-in supports "downloading-keys" and "downloading roles"

• unity-py properly supports asset role (https://github.com/radiantearth/stac-spec/blob/master/item-spec/item-spec.md#asset-roles )

• Existing Workflows are updated to use new stage-in/out containers

• Create validation STAC outputs from unity-py

• file a ticket with CMR to use the asset-role in their results"

Work Tickets

• Support asset "roles" for STAC in Data Catalog data Searches unity-py#67
• STAC Collection Utils to support asset-role  unity-py#69
• Update UDS STAC response to conform to new asset schema unity-data-services#287
• Update stage-in to key off either roles or keys unity-data-services#288
• Update translation between STAC and CNM in catalog step unity-data-services#293
• Update stage out to key off of asset role unity-data-services#296
• Update stage-in, stage-out versions and environments to support new STAC metadata requirements app-pack-generator#20
• Update sample applicaiton template with updated unity-py version
• re-produce application packages with updated
• Implement asset-role for STAC results nasa/Common-Metadata-Repository#2045

Dependencies

• TBD

Risks

• STAC backwards compatibility #129

[mike-gangl]

Need to have a discussion with @rtapella and @ngachung on the timing of this release.

• Timing of deployment of DAPA Search result updates
• Timing of deployment of DAPA ingest updates
• Timing of availability of new Stage-in tasks
• Timing of availability of new Stage-out tasks
• Timing of availability of unity-py Stac utilities updates
• Update application packages with unity py updates, stage-in, out docker containers

[GodwinShen]

@ngachung which work tickets are associated with the acceptance criteria shown below?

• STAC asset key is not based on "data" or "metadata" - should be unique and not known ahead of time and based on filename
• Ingest commands can ingest multiple "data" role types successfully
• file a ticket with CMR to use the asset-role in their results"

[GodwinShen]

@mike-gangl is this something that you would do? If so, I guess we should have a ticket for it if you don't already have one elsewhere: Existing Workflows are updated to use new stage-in/out containers

[GodwinShen]

@rtapella , @anilnatha do we have a ticket for this item yet? "Create validation STAC outputs from unity-py". The work tickets listed here don't immediately look applicable to me, but that could just be my lack of understanding.

[rtapella]

Good question. I am not sure what that means exactly or who would do it. Does it mean testing that this works? (specifically taht tickets 67 and 69 in unity.py work, the first two checkboxes)

[GodwinShen]

@mike-gangl what do we mean by the task "Create validation STAC outputs from unity-py"? I don't know if this is something we need UI/UX to do or what. Can you help clarify this one?

[mike-gangl]

Open question- how can we inform U-DS which file is the stac-json file to read metadat from. Consider the case where we have multiple metadata file ending with .json?

data_file.tiff (role: data)
data_file.met.json (role: metadata)
data_file.json (role: metadata, stac, u-ds needs this one)
data_file.mdf (role: metadata)
data_file.png (role: browse)

I wonder if we should call the file data_file.stac.json so we have a good indication of what to key off of? in this case, we can set the cumulus configuration to look for *.stac.json files

@ngachung thoughts?

[ngachung]

We can standardize on data_file.stac.json, but for now our implementation as described in unity-sds/unity-data-services#301 is to parse all the json files until we get a valid stac feature.

[mike-gangl]

@GodwinShen - regarding "Validation Outputs" means to work with U-DS to ensure the output files from to_stac() are valid from the U-DS perspective. This is for stage-out and the ingestion portions of stac.

[rtapella]

marked Done but never Closed