variables_as_arguments results in a large number of variable queries

[cpboyd]

I've been tracing an issue where our Airflow instances are querying our secrets backends thousands of times per minute. The main requests seem to be specified under variables_as_arguments.

From https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html:

In top-level code, variables using jinja templates do not produce a request until a task is running, whereas, Variable.get() produces a request every time the dag file is parsed by the scheduler. Using Variable.get() will lead to suboptimal performance in the dag file processing. In some cases this can cause the dag file to timeout before it is fully parsed.

variables_as_arguments queries each variable with Variable.get() twice (if the variable exists):

dag-factory/dagfactory/dagbuilder.py

Lines 557 to 561 in 7d2782b

	for variable in variables:
	if Variable.get(variable["variable"], default_var=None) is not None:
	task_params[variable["attribute"]] = Variable.get(
	variable["variable"], default_var=None
	)

Are there any alternatives to using the variables_as_arguments?