You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When I try the feature of running jobs on the job cluster instead of the all-purpose cluster, both the purpose cluster and the job cluster trigger, and only when the all-purpose starts, and the the job cluster follows.
Steps To Reproduce
Configure all purpose cluster detail in profiles.yaml
Once configured job cluster detail provided, all purpose cluster should not be trigged.
Screenshots and log output
System information
The output of dbt --version:
Core:
- installed: 1.7.3
- latest: 1.7.7 - Update available!
Your version of dbt-core is out of date!
You can find instructions for upgrading here:
https://docs.getdbt.com/docs/installation
Plugins:
- databricks: 1.7.2 - Update available!
- spark: 1.7.1 - Up to date!
At least one plugin is out of date or incompatible with dbt-core.
You can find instructions for upgrading here:
https://docs.getdbt.com/docs/installation
The operating system you're using:
Mac OS, should be irrelevant
The output of python --version:
Python 3.10.13
Additional context
NaN
The text was updated successfully, but these errors were encountered:
This is expected behavior, as python models are integrated into the rest of your dbt project using SQL (for example, on an incremental model, the merge behavior is conducted in SQL), and that SQL would be executed on the AP Cluster. We are investigating ways for python model behavior to be more 'spark-like', but for now I would say this is an enhancement request, rather than a bug, as it is consistent with the structure imposed by dbt-core.
I am not sure but it looks to me, that the strict seperation between execution (dbt python code) and the model execution (putting model into an isolated space) seems to be a bit oversized on Databricks job clusters, because the job will run nevertheless on spark on the master node. But maybe I am not getting the full picture of this issue...
Describe the bug
When I try the feature of running jobs on the job cluster instead of the all-purpose cluster, both the purpose cluster and the job cluster trigger, and only when the all-purpose starts, and the the job cluster follows.
Steps To Reproduce
profiles.yaml
dbt_project.yml
, follow instructions (https://docs.getdbt.com/docs/build/python-models#specific-data-platforms)Expected behavior
Once configured job cluster detail provided, all purpose cluster should not be trigged.
Screenshots and log output
System information
The output of
dbt --version
:The operating system you're using:
Mac OS, should be irrelevant
The output of
python --version
:Python 3.10.13
Additional context
NaN
The text was updated successfully, but these errors were encountered: