Triggerer process does not reload trigger implementations after git-sync when deployed on kubes

[JossWhittle]

Consider the scenario:

• Airflow deployed on kubernetes
• with git-sync sidecars to load the DAGs into the airflow containers
• and custom Trigger classes implemented alongside DAGs which are being git-synced.

A DAG using a custom Sensor with accompanying custom Trigger is already running, and the Airflow triggerer container has spawned, git-cloned, and started running the custom Trigger.

• Now a change to the Trigger implementation is pushed to the remote git repo.

The currently running Trigger continues to execute as expected. It responds correctly to it's event and the deferred task is resumed correctly.

• The DAG is run again, and reaches the Sensor task
• it correctly uses any newly git-synced code for the DAGs and tasks executed on kubes pod workers

But when the Sensor defers and the triggerer container picks up the request to run the custom Trigger it reuses the original implementation.

Getting the custom Trigger to refresh to the latest code requires killing the triggerer container and letting kubes recreate it.

My current theories are:

• The custom Trigger is imported and stored in memory in the async event loop process that is still running in the triggerer container
• Serialized version of the custom Trigger is exported from the scheduler container using the newly git-synced code
• and is then being deserialized in the triggerer container using the original first git-cloned code.

This feels like a bug. I'd love to get other peoples inputs here.

Can it be made safe to implement deferable Sensors and Triggers in git-synced code?

[potiuk]

This is as expected. Triggerer does not reload Triggers dynamically, it needs to be restarted. PRs to change the behaviour are welcome.

[JossWhittle]

Is it desirable to change the behavior? Or should it be documented that custom Triggers + git-sync is not safe?

What would be the best practice mechanism for updating Trigger implementations? Force a redeployment by baking them into an updated container and bumping the image tag on the chart?

[potiuk]

Is it desirable to change the behavior? Or should it be documented that custom Triggers + git-sync is not safe?

Feel free to document it if you are not up to the PR/proposal to change it. You are probably the best person to provide PR and explain it. You cna then join > 2500 contributors who (most of them) found something that needs to be improved and contributed it. This is also a great way to give back to the community for the free software you get from - mostly - private time of those contributors.

What would be the best practice mechanism for updating Trigger implementations? Force a redeployment by baking them into an updated container and bumping the image tag on the chart?

Restart triggerer when trigger change. You can do it in various ways. A nice description about options you can think of would also be great as documentation.