Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Have SirCAL use the "Active" flag from the API metadata #1961

Open
melange396 opened this issue May 10, 2024 · 0 comments
Open

Have SirCAL use the "Active" flag from the API metadata #1961

melange396 opened this issue May 10, 2024 · 0 comments
Labels
data quality Missing data, weird data, broken data devops Engineering Used to filter issues when synching with Asana future-solution Solutions to problems we don't have yet but still dread good first issue

Comments

@melange396
Copy link
Contributor

Sir Complains-a-Lot lets us know about potential data pipeline problems by noticing when our indicator signals havent been properly updated for a while. Sometimes a signal may be discontinued for any number of reasons, and it will stop getting updated with new data. To prevent SirCAL from alerting on such signals, it consults a list of "retired" signals. Unfortunately, maintaining this list requires enumerating them all by hand (such as here and here) and the list will only grow larger over time.

Instead of relying [solely?] on the "retired" signal list, SirCAL can make use of the Active status in the metadata available through the Delphi Epidata API to determine if a signal is worth alerting on -- if a signal is not marked as being "Active", it is no longer expected to receive updates and it can thus be ignored.

A caveat to this is that SirCAL currently uses the covidcast client package to access metadata. That client uses the /covidcast_meta endpoint, but the "Active" status is only available from an alternate endpoint, /covidcast/meta.

@melange396 melange396 added data quality Missing data, weird data, broken data good first issue Engineering Used to filter issues when synching with Asana future-solution Solutions to problems we don't have yet but still dread devops labels May 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
data quality Missing data, weird data, broken data devops Engineering Used to filter issues when synching with Asana future-solution Solutions to problems we don't have yet but still dread good first issue
Projects
None yet
Development

No branches or pull requests

1 participant