Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enforce metric data parity between Snowflake and DynamoDb #1198

Open
1 of 2 tasks
manasaV3 opened this issue Aug 2, 2023 · 0 comments
Open
1 of 2 tasks

Enforce metric data parity between Snowflake and DynamoDb #1198

manasaV3 opened this issue Aug 2, 2023 · 0 comments
Assignees
Labels

Comments

@manasaV3
Copy link
Collaborator

manasaV3 commented Aug 2, 2023

As the hub is not fetching data directly from GitHub/ PyPi and is relying on the snowflake data, we treat snowflake data as the single source of truth, and the data in dynamo should maintain parity with snowflake.

In the cases where data gets added/removed from snowflake tables/views that the hub consumes from, for any reason, we should be able to update the hub data to reflect that. We need an automated workflow to validate that our data is in sync with Snowflake and heal the discrepancies.

This workflow could even be set to run periodically to ensure data quality.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: Backlog
Development

No branches or pull requests

1 participant