Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pyarrow/pandas: add load id and dlt id in the extract phase and unify the behavior #1317

Open
rudolfix opened this issue May 3, 2024 · 0 comments
Assignees
Labels
tech-debt Leftovers from previous sprint that should be fixed over time

Comments

@rudolfix
Copy link
Collaborator

rudolfix commented May 3, 2024

Background
By default we do not add load_id and dlt_id to arrow tables. This must be configured explicitly and happens in the normalizer.
As a consequence, we need to decompress and rewrite parquet files which takes a lot of resources.
In this ticket we move this behavior to the extract phase. This is against general architecture but I do not see any other way to do that without rewriting files.
We also unify the behavior making relational normalizer to follow ItemsNormalizerConfiguration

Implementation
We split this ticket into several PR.
PR 1.

    • add load_id in the extract phase.
    • make sure we do not clash with normalize which also add load_id (can we remove it from there?)
    • we (probably) do not need the logic that adds the columns when writing a file. we can just add them to existing table
    • ItemsNormalizerConfiguration must be taken into account. this is probably a breaking change because we need to move it from normalize to extract so old settings will stop working. or maybe you'll find a clever solution here :)

PR 2.

    • add dlt_id in the extract phase
    • when adding _dlt_id we must follow table settings and generate _dlt_id according to hints (ie. SCD2 look how relational.py generates different hashes.). also we have a fast method to generate content hashesh: add_row_hash_to_table
    • observe "bring your own hash". if there's a column with unique, do not add _dlt_id (random one). if we have SCD2 type hash (please see SCD2 documentaiton on how to add it) we also skip it

PR 3.
Unify relational and pyarrow behavior:

    • in relational.py do not generate _load_id and _dlt_id if not switched on (ItemsNormalizerConfiguration)
    • observe bring your own hash in relational.py. please note that now we use _dlt_id if it is present in the data. way better would be to detect unique column in schema and force to use it. same for SCD2

PR 4.
Unify behavior 2

    • in _compute_table (extract / pyarrow) when we add new columns we should also infer hints like for any new columns. currently schema settings will be ignored
@rudolfix rudolfix added the tech-debt Leftovers from previous sprint that should be fixed over time label May 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
tech-debt Leftovers from previous sprint that should be fixed over time
Projects
Status: Planned
Development

No branches or pull requests

2 participants