Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Write-Audit-Publish pattern support #903

Open
dwreeves opened this issue Mar 27, 2024 · 1 comment
Open

[Feature] Write-Audit-Publish pattern support #903

dwreeves opened this issue Mar 27, 2024 · 1 comment
Labels
area:parsing Related to parsing DAG/DBT improvement, issues, or fixes dbt:run Primarily related to dbt run command or functionality execution:local Related to Local execution environment parsing:custom Related to custom parsing, like custom DAG parsing, custom DBT parsing, etc
Milestone

Comments

@dwreeves
Copy link
Collaborator

It should be possible to do something like this but with Cosmos's DbtOperators:

set -e
dbt run --target blue
dbt test --target blue
cp target/manifest.json build-artifacts/manifest.json
dbt clone --target green --state build-artifacts/manifest.json --full-refresh

Basically, a write-audit-publish pattern. Right now, DbtDags are able to conceptualize a node as a TaskGroup; here the task group would consist of, essentially run -> test -> clone.


This is mildly related to #895, as this should be a pattern that is reasonably achievable for users to implement via the middle API layer. I think though, in the longer-term, this should be explicitly supported as a feature in Cosmos.

Copy link

dosubot bot commented Mar 27, 2024

It seems like you've got this under control, if you want help or have specific questions, let me know what I can do for you!

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

@dosubot dosubot bot added area:parsing Related to parsing DAG/DBT improvement, issues, or fixes dbt:run Primarily related to dbt run command or functionality execution:local Related to Local execution environment parsing:custom Related to custom parsing, like custom DAG parsing, custom DBT parsing, etc labels Mar 27, 2024
@tatiana tatiana added this to the 1.6.0 milestone May 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:parsing Related to parsing DAG/DBT improvement, issues, or fixes dbt:run Primarily related to dbt run command or functionality execution:local Related to Local execution environment parsing:custom Related to custom parsing, like custom DAG parsing, custom DBT parsing, etc
Projects
None yet
Development

No branches or pull requests

2 participants