You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Inspired by dlt schema features (auto schema, schema evolution, data contracts), we could create lifecycle adapter to capture the schema of dataframe objects and store them to a file (pandas, polars, pyarrow, and Ibis to get started?).
Then, the same adapter could do subsequent runs:
strict check: fail if schema doesn't match
warning: try to execute, but log schema mismatch
evolution: update the stored schemas
We could possibly convert these schemas to dlt schemas for full visibility over ETL. Similarly, Ibis expression allows for granular lineage (e.g., "new_users" from HamiltonNode 1 was renamed to "users" in HamiltonNode 3)
Describe the solution you'd like
Use lifecycle adapters, in particular run_after_node_execution and run_after_graph_execution. The challenge is to find the right format to store this information and integrate nicely with other tools
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
The text was updated successfully, but these errors were encountered:
Inspired by dlt schema features (auto schema, schema evolution, data contracts), we could create lifecycle adapter to capture the schema of dataframe objects and store them to a file (pandas, polars, pyarrow, and Ibis to get started?).
Then, the same adapter could do subsequent runs:
We could possibly convert these schemas to dlt schemas for full visibility over ETL. Similarly, Ibis expression allows for granular lineage (e.g., "new_users" from HamiltonNode 1 was renamed to "users" in HamiltonNode 3)
Describe the solution you'd like
Use
lifecycle
adapters, in particularrun_after_node_execution
andrun_after_graph_execution
. The challenge is to find the right format to store this information and integrate nicely with other toolsDescribe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
The text was updated successfully, but these errors were encountered: