-
Notifications
You must be signed in to change notification settings - Fork 14
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Lake][DuckDB] ETL - Implement Update Queries #1001
Comments
To implement this ticket, we should first start w/ simply updating predictions when truevals and payouts show up [How this ticket grows] In the future...
subscription event slot event prediction event trueval event payout event |
[Feedback Mustafa]
[Effective Processing of Events]
[Simplify Requirements even Further] Trueval does not contain the user id anywhere, so it cannot update the prediction table directly. Literally, all we need is to join payouts with predictions. |
Motivation
We have now verified that the basic lake functionality is working as expected.
We now want to verify the data quality and completeness.
This means that additional SQL queries are being run, such that more tables are being processed and richer data is being generated.
Update Step - Incrementally updating the Lake
When you run the "lake update" command, later SQL queries are responsible for updating w/ the most recent information.
Data Workflows
All data workflows should operate in the same way.
DoD:
Task:
The text was updated successfully, but these errors were encountered: