Change how the profiles.yml is created when using mocked profiles

[tatiana]

Context

As of Cosmos 1.3, when users use Cosmos ProfileMapping we create the profiles.yml file in two different ways:

• functional: mapping keys to environment variables (necessary to run dbt commands that need to access the database). Example:
  

  astronomer-cosmos/cosmos/profiles/postgres/user_pass.py

  Line 39 in d1dbc41

  def profile(self) -> dict[str, Any | None]:

• mocked: mapping keys to hard-coded fake values (to parse the dbt project when using dbt ls without the need to run dbt commands that reach the database). Example:
  

  astronomer-cosmos/cosmos/profiles/postgres/user_pass.py

  Line 55 in d1dbc41

  def mock_profile(self) -> dict[str, Any | None]:

  
  

  astronomer-cosmos/cosmos/profiles/base.py

  Line 174 in d1dbc41

  def mock_profile(self) -> dict[str, Any]:

The mocked approach was introduced so users could validate their DAGs in their CI without creating Airflow connections and without exposing database credentials.

The challenge with this approach is that, when using partial parsing, dbt expects the profiles.yml to remain unchanged, as observed during #904. On the positive side, dbt is happy to use partial parse if only the content of the environment variable changes (validated with dbt 1.7.10).

This PR #904 worked around the issue by allowing users to leverage partial parsing to avoid creating mocked profiles (using RenderConfig(enable_mock_profile=False)). However, users must ensure the CI has the desired connection (potentially with fake values). The goal of this ticket is to overcome this issue.

Acceptance criteria

• Cosmos should create the dbt profiles.yml in the same way, referencing environment variables
• When using mock, Cosmos should use mocked values in the environment variables
• Users should still be able to parse the Cosmos DAG that uses ProfileMapping by running airflow dags list when no connection was created
• Cosmos users should be able to use partial parsing during DAG building with dbt ls even when not usingRenderConfig(enable_mock_profile=False).

To validate the last criteria, the developer can change the example DAG dev/dags/basic_cosmos_task_group.py, by removing enable_mock_profile=False, run it twice with a command similar to:

AIRFLOW__COSMOS__CACHE_DIR=/tmp/
AIRFLOW__COSMOS__ENABLE_CACHE=1 AIRFLOW__LOGGING__LOGGING_LEVEL=DEBUG AIRFLOW__COSMOS__PROPAGATE_LOGS=0 AIRFLOW_CONN_AIRFLOW_DB="postgres://postgres:postgres@0.0.0.0:5432/postgres" PYTHONPATH=`pwd` AIRFLOW_HOME=`pwd` AIRFLOW__CORE__DAGBAG_IMPORT_TIMEOUT=20000 AIRFLOW__CORE__DAG_FILE_PROCESSOR_TIMEOUT=20000 airflow dags test basic_cosmos_task_group  `date -Iseconds`

And confirm that the second time, it does not print "Unable to do partial parsing"

[pankajastro]

We tested it, but unfortunately, the partial parsing does not work even when the environment variable value changes. In the example below, I change the POSTGRES_HOST environment variable value, but partial parsing still does not work.

(devenv) pankaj@Pankajs-MacBook-Pro astronomer-cosmos % cat .env 
POSTGRES_HOST=0.0.0.0
POSTGRES_USER=postgres
POSTGRES_PORT=5432
POSTGRES_DB=postgres
POSTGRES_SCHEMA=public

(devenv) pankaj@Pankajs-MacBook-Pro astronomer-cosmos % source .env 

(devenv) pankaj@Pankajs-MacBook-Pro astronomer-cosmos % dbt ls --target-path dev --profiles-dir dev/dags/dbt/perf/ --project-dir dev/dags/dbt/perf/
20:12:55  Running with dbt=1.8.1
20:12:55  Registered adapter: postgres=1.8.0
Manifestpath:  dev/dags/dbt/perf/dev/partial_parse.msgpack
20:12:56  Unable to do partial parsing because profile has changed
20:12:56  Found 413 macros
20:12:56  No nodes selected!

(devenv) pankaj@Pankajs-MacBook-Pro astronomer-cosmos % dbt ls --target-path dev --profiles-dir dev/dags/dbt/perf/ --project-dir dev/dags/dbt/perf/
20:13:01  Running with dbt=1.8.1
20:13:01  Registered adapter: postgres=1.8.0
Manifestpath:  dev/dags/dbt/perf/dev/partial_parse.msgpack
20:13:01  Found 413 macros
20:13:01  No nodes selected!

(devenv) pankaj@Pankajs-MacBook-Pro astronomer-cosmos % source .env                                                                                

(devenv) pankaj@Pankajs-MacBook-Pro astronomer-cosmos % cat .env                                                                                   
POSTGRES_HOST=localhost
POSTGRES_USER=postgres
POSTGRES_PORT=5432
POSTGRES_DB=postgres
POSTGRES_SCHEMA=public

(devenv) pankaj@Pankajs-MacBook-Pro astronomer-cosmos % dbt ls --target-path dev --profiles-dir dev/dags/dbt/perf/ --project-dir dev/dags/dbt/perf/
20:13:30  Running with dbt=1.8.1
20:13:30  Registered adapter: postgres=1.8.0
Manifestpath:  dev/dags/dbt/perf/dev/partial_parse.msgpack
20:13:31  Unable to do partial parsing because profile has changed
20:13:31  Found 413 macros
20:13:31  No nodes selected!

(devenv) pankaj@Pankajs-MacBook-Pro astronomer-cosmos % cat dev/dags/dbt/perf/profiles.yml 
default:
  target: dev
  outputs:
    dev:
      type: postgres
      host: "{{ env_var('POSTGRES_HOST') }}"
      user: "{{ env_var('POSTGRES_USER') }}"
      password: "{{ env_var('POSTGRES_PASSWORD') }}"
      port: "{{ env_var('POSTGRES_PORT') | int }}"
      dbname: "{{ env_var('POSTGRES_DB') }}"
      schema: "{{ env_var('POSTGRES_SCHEMA') }}"
      threads: 4
(devenv) pankaj@Pankajs-MacBook-Pro astronomer-cosmos %