Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Lake][DuckDB] Stabilize pdr lake raw + etl update in duckdb-integration PR #1036

Closed
5 tasks done
idiom-bytes opened this issue May 14, 2024 · 4 comments
Closed
5 tasks done
Assignees
Labels
Type: Enhancement New feature or request

Comments

@idiom-bytes
Copy link
Member

idiom-bytes commented May 14, 2024

Background / motivation

Right now, in duckdb latest w/ validations, the lake raw update command fails on pdr-slots.
image

TODOs / DoD

Fix pdr-slots and make sure that pdr lake update command and pdr etl update command are completed e2e, and that validations or other problems are not stopping the pipeline.

Tasks:

  • Fix validation, schema, or new system problem that's causing GQLDF steps to fail.
  • pdr lake update command and pdr etl update command are completing e2e
  • GQLDF pdr-slots is running, problem is addressed/stability improved
  • pdr-slots and bronze-slots "should just work" (and were working before the validation merge)
  • Merge into duckdb-integration PR
@KatunaNorbert
Copy link
Member

GQLDF pdr-slots + bronze-slots are finally, disabled for now (they were just working, so let's fix them and understand why they broke)

I don't understand this points, should we disabled this tables?

@idiom-bytes
Copy link
Member Author

idiom-bytes commented May 14, 2024

I don't understand this points, should we disabled this tables?

Now: We prioritised this fix mainly because the validation update broke it...

Next: I want to turn off these extra tables and only have the stuff required for #1000 so we can "freeze a release candidate".

Right After: We can work towards the next release candidates, enabling more tables and SQL queries.

@idiom-bytes
Copy link
Member Author

idiom-bytes commented May 14, 2024

I have just tested the bronze_slots table and as far as I can tell, it is not working.

If I remember correctly, it was working prior to the validation update

@KatunaNorbert KatunaNorbert self-assigned this May 15, 2024
@idiom-bytes
Copy link
Member Author

I have been playing with the latest, and they are working.
I'm still not confident they are correct, but they are building.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Type: Enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants