Using MetastoreBackend in hooks triggers CommitProhibitorGuard

[Blizzke]

Apache Airflow version

2.8.2

If "Other Airflow 2 version" selected, which one?

No response

What happened?

We have a number of "@hookimpl"-hooks that run to send out messages to chat channels when specific things happen (for example an on_dag_run_failed that notifies people).
For that, a webhook-url is needed, which is obtained from the variables (as it can differ per DAG and is controlled by our developers themselves, just to say that the config file is not an option here, nor is the environment).

Because the hook is triggered from within "stuffs" that happen in the scheduler, it runs in a CommitProhibitorGuard-context. The MetastoreBackend uses the @provide_session-decorator, which tries to commit at the end and triggers an exception.

I've been very hopefully watching #12818 (and #21283) but unfortunately it seems they have stalled development wise.
For now we had to resolve things by duplicating the Variable.get_variable_from_secrets() implementation, but specifying a throwaway session for the backends that support it. Obviously I would love to get rid of this again asap.
I understand that this is probably a very niche situation, but I'm adding this issue to point out that that decorator is causing problems elsewhere too.

What you think should happen instead?

Should be able to obtain a variable value in the normal way.

How to reproduce

Basically use Variable.get() from within a hook callback.

Operating System

linux

Versions of Apache Airflow Providers

No response

Deployment

Docker-Compose

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

This is an interesting discussion, but a discussion - not an issue.

If I may suggest - rather than observing, I suggest you or your team contributes a change to fix it. Starting from the PRs you saw - leading that to completion is exactly how things are done in Airflow. You seem to need it, so conttributing a change back tha enables the behaviour you mentioned seems like an obvious way of solving it. Isn't it ?

[ChrnvaN]

@Blizzke Hello!
Did you manage to set up the correct reading of values from Variable when on_dag_run_failed?

[Blizzke]

I mentioned it in my original post: we duplicated the Variable.get_variable_from_secrets() function with it's own session, you can use the provide_session decorator for that.

[ChrnvaN]

Can you show me what it looks like please?