Enable detection of errors in Variable.get() such as absence of key method and connection failure to Metastore

[tyama711]

Description

When attempting to retrieve a value by specifying a key using the Variable.get() method, there are cases where the retrieval fails, such as:

• When the specified key does not exist in environment variables or Metastore
• When there is a connection failure to Metastore or some other error

However, in the current implementation of Airflow, all exception handling is being suppressed in the following part, making it impossible for users to differentiate these cases:

airflow/airflow/models/variable.py

Lines 286 to 291 in defe459

	except Exception:
	log.exception(
	"Unable to retrieve variable from secrets backend (%s). "
	"Checking subsequent secrets backend.",
	type(secrets_backend).__name__,
	)

I believe that Variable.get() should return None if the specified key does not exist, and it should raise an exception for unexpected errors such as connection errors.

Use case/motivation

With this feature, it becomes possible to handle errors specific to each cause as follows:

• When the specified key does not exist in environment variables or Metastore --> Use some default value
• When there is a connection failure to Metastore or some other error --> Retry several times, and if unsuccessful, fail the job.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[romsharon98]

Hi @tyama711 thanks for opening this issue.
You can give the get function a default value if the variable is not exist.

Variable.get(key='dummy_key', default_var='default_value')

As you said you can retry for any exception you get so I don't think we need to return more specific exception, let me know what you think.

[github-actions]

This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.