Speed up dbt run "pre-query" process for a large project

[thomasaarholt]

We have what I suspect is a medium-large dbt project, with just over 2000 models.
Every time I run dbt run -s mymodel, it takes 7 seconds before my query (or pre-hooks) start running. This isn't a major issue, but it would be nice if this section could be "cached" or similar, especially as the tooling (like vscode extensions) is getting better and better, and more processes run the equivalent of dbt run.

Here's a snippet from my terminal:

❯ dbt run -s intm_dynamic_storage_location_popularity --target prod
10:04:05  Running with dbt=1.6.6
10:04:06  Registered adapter: snowflake=1.6.4
10:04:09  Found 2208 models, 98 snapshots, 19 analyses, 2958 tests, 22 seeds, 2 operations, 945 sources, 0 exposures, 9 metrics, 1029 macros, 0 groups, 1 semantic model
10:04:09
10:04:12
10:04:12  Running 2 on-run-start hooks
(...)

Note the timestamps, we start at :05 and aren't doing real work before ":12". If this section could be sped up, especially on repeated runs of the same model, I think it would make dbt feel much snappier.

Does anyone have any tips on how I can improve performance of this part?

[ttusing]

Could check out https://docs.getdbt.com/reference/global-cli-flags#record-timing-info to identify what process is slowest and see if its possible to optimize it

[thomasaarholt]

Thanks. Sorry for the very late reply. This was an interesting approach. 72 of the 82 seconds of the attached run (one model) seems to be "~:0(<method 'acquire' of '_thread.lock' objects>)". Can't say I fully understand that.

timing.html.zip

[jkleinkauff]

@thomasaarholt try to execute your model with dbt --log-level debug run. Maybe it could help to track what dbt is doing before the model execution. Even before the model execution, dbt will run some metadata listing on your db, it could be it.