You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently Airflow offers failed-deps to investigate why task isn't being scheduled. This is very helpful tool however it works only in real time according to the current entries in the metadb. Investigating past anomalies isn't supported.
Sometimes scheduling problems are "solved" on their own. It could be that pool is overcrowded or concurrency has been reached but eventually stress is reduced and tasks are scheduled, thus when you notice it and want to investigate why there was a delay to begin with your capabilities are limited as there could be many reasons.
The needed solution:
We should investigate the option to audit the failed-deps information or alternatively offer an easy way to export this information in real time to an external audit storage for later investigation.
Committer
I acknowledge that I am a maintainer/committer of the Apache Airflow project.
The text was updated successfully, but these errors were encountered:
Recently I worked on this and the information is available as part of UI and API for tasks in scheduled or None state. Perhaps the API could be used for export and also enriched with additional checks that provide useful information.
Recently I worked on this and the information is available as part of UI and API for tasks in scheduled or None state. Perhaps the API could be used for export and also enriched with additional checks that provide useful information.
Body
Currently Airflow offers failed-deps to investigate why task isn't being scheduled. This is very helpful tool however it works only in real time according to the current entries in the metadb. Investigating past anomalies isn't supported.
Sometimes scheduling problems are "solved" on their own. It could be that pool is overcrowded or concurrency has been reached but eventually stress is reduced and tasks are scheduled, thus when you notice it and want to investigate why there was a delay to begin with your capabilities are limited as there could be many reasons.
The needed solution:
We should investigate the option to audit the
failed-deps
information or alternatively offer an easy way to export this information in real time to an external audit storage for later investigation.Committer
The text was updated successfully, but these errors were encountered: