Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Table from SQLAlchemy/Pydantic model #2150

Open
9 tasks
1cadumagalhaes opened this issue May 3, 2024 · 1 comment
Open
9 tasks

Table from SQLAlchemy/Pydantic model #2150

1cadumagalhaes opened this issue May 3, 2024 · 1 comment
Labels
feature New feature or request

Comments

@1cadumagalhaes
Copy link

1cadumagalhaes commented May 3, 2024

Please describe the feature you'd like to see
I want to be able to create a Table from a SQLAlchemy model, import it's columns with types and constraints.
Context: I'm running astro on a database from which the tables and migrations are defined by SQLALchemy models, that are also used by the application.

What happens here is:

  1. I declare my tables as models and write the database migrations based on it.
  2. The data for some of these tables come from aql.transform (with replace or append) and aql.merge.
  3. If I just declare the tables with the name, conn_id and metadata, when replacing the table it will lose all constraints, and might get some types wrong. One common issue with this behaviour is that tables lose their primary key, but also all foreign keys, sequences and etc.
  4. On the other hand, if I already have my tables defined as classes, it should not be hard to get every definition about it

Describe the solution you'd like

I'm not sure about the syntax, but we could do something like:

Table(model=MySQLAlchemyClass, conn_id=connection_id)

# Or even

Table(conn_id=connection_id).from_model(MySQLAlchemyClass)

With this, we could import the table name, the schema, all columns with types and constraints

Are there any alternatives to this feature?
I don't think so

Additional context
There is also sqlmodel which adds annotations to models so they're compatible with both pydantic and SQLAlchemy.

But adding support to python dataclasses would be nice for someone who isn't using SQLAlchemy or a similar ORM, but still wants an easier way of defining their table specifics.

Acceptance Criteria

  • All checks and tests in the CI should pass
  • Unit tests (90% code coverage or more, once available)
  • Integration tests (if the feature relates to a new database or external service)
  • Example DAG
  • Docstrings in reStructuredText for each of methods, classes, functions and module-level attributes (including Example DAG on how it should be used)
  • Exception handling in case of errors
  • Logging (are we exposing useful information to the user? e.g. source and destination)
  • Improve the documentation (README, Sphinx, and any other relevant)
  • How to use Guide for the feature (example)
@1cadumagalhaes 1cadumagalhaes added the feature New feature or request label May 3, 2024
@1cadumagalhaes
Copy link
Author

For now I'll try to create a function/decorator to do this conversion and if it works I'll add it here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant