Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Read multiple YAML files in the same python file #122

Open
Ferdinanddb opened this issue Jan 12, 2022 · 0 comments
Open

Read multiple YAML files in the same python file #122

Ferdinanddb opened this issue Jan 12, 2022 · 0 comments

Comments

@Ferdinanddb
Copy link

Hello,

This is not really an issue but more a question, I am wondering if you would recommend to have the following : instead of having one python file that would read one yaml file and construct a DAG, you construct multiple DAGs in the same python file by reading several YAML files.

From what I saw, the bottleneck would be at the worker level because it also has to parse the DAG to understand what it needs to do, and thus in this situation the worker would parse all the YAML files while it only needed to parse one. I did not find out if there is a way to stipulate that information to the airflow worker to not having it to parse something it doesn't need.

Regards
Ferdinand

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant