New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Newly created and approved DAG takes time to load in the DagBag #1

Open

shivanshs9 opened this issue Aug 15, 2020 · 2 comments

Contributor

shivanshs9 commented Aug 15, 2020

Implement a way to force sync the new DAG in the DagBag

Contributor Author

shivanshs9 commented Aug 15, 2020

Current Blockers:

Not possible to update in-memory instances of DagBag of all the workers. Right now, the only way (which is also unreliable) is to wait for workers to refresh on their own according to Airflow configs (worker_refresh_batch_size and worker_refresh_interval).

Contributor Author

shivanshs9 commented Aug 15, 2020

Airflow issue - apache/airflow#10341

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment