[Bug] dbtCore - after upgrading, can't run DBT

[moseleyi]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

I installed dbt-core, bigquery, and postgres, and then when typing dbt --version I get this error:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\___\AppData\Roaming\Python\Python312\Scripts\dbt.exe\__main__.py", line 4, in <module>
  File "C:\Users\___t\AppData\Roaming\Python\Python312\site-packages\dbt\cli\__init__.py", line 1, in <module>
    from .main import cli as dbt_cli  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\___\AppData\Roaming\Python\Python312\site-packages\dbt\cli\main.py", line 14, in <module>
    from dbt.cli import requires, params as p
  File "C:\Users\___\AppData\Roaming\Python\Python312\site-packages\dbt\cli\requires.py", line 10, in <module>
    from dbt.adapters.factory import adapter_management, register_adapter, get_adapter
ModuleNotFoundError: No module named 'dbt.adapters.factory'

This is after installing bigquery, postgres,

Expected Behavior

Should show the version of dbt packages

Steps To Reproduce

1. Install dbt-core
2. Run dbt --version

Relevant log output

No response

Environment

- OS: Windows
- Python: 3.12
- dbt: 1.8

Which database adapter are you using with dbt?

No response

Additional Context

No response

[jtcohen6]

@moseleyi Could you run pip freeze | grep 'dbt' and report back with the exact versions of Python packages that you have installed?

Also - you might try installing all of these by name, along with the --upgrade flag, and see if that resolves the issue for you:

pip install --upgrade dbt-core dbt-bigquery dbt-postgres

[rubenhelsloot]

@jtcohen6 I have the same issue, this is the output you asked for:

❯ pip freeze | grep 'dbt'
dbt-adapters==1.1.1
dbt-common==1.0.4
dbt-core==1.8.0
dbt-extractor==0.5.1
dbt-postgres==1.8.0
dbt-semantic-interfaces==0.5.1
sqlfluff-templater-dbt==3.0.6

The issue persists when I uninstall and then reinstall dbt-core and dbt-postgres. FWIW I'm using poetry.

[jtcohen6]

Thanks @rubenhelsloot! Are you also using Python 3.12?

[moseleyi]

My output:

dbt-adapters==1.1.1
dbt-common==1.0.4
dbt-core==1.8.0
dbt-extractor==0.5.1
dbt-postgres==1.8.0
dbt-semantic-interfaces==0.5.1

Some people in my team got it working but maybe because it was clean install, they never had to use python packages before. Perhaps some dependencies of dbt are not updated with the latest update?

[moseleyi]

Surprisingly dbt 1.7.13 works fine with Python 3.12.3 - I checked on my other laptop

[dbeatty10]

@rubenhelsloot! Are you also using Windows?

@moseleyi Which commands are you using to installing dbt? Are you using a virtual environment with venv or Poetry?

The following files and commands worked for me:

• on macOS
• using venv
• with Python 3.12.3 (managed by pyenv)

Create this file:

requirements.txt

dbt-adapters==1.1.1
dbt-common==1.0.4
dbt-core==1.8.0
dbt-extractor==0.5.1
dbt-postgres==1.8.0
dbt-semantic-interfaces==0.5.1

Run these commands:

python3 -m venv env
source env/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt
source env/bin/activate
python --version
dbt --version
deactivate

Output:

Python 3.12.3
Core:
  - installed: 1.8.0
  - latest:    1.8.0 - Up to date!

Plugins:
  - postgres: 1.8.0 - Up to date!

[leoebfolsom]

In case helpful, when I upgraded to dbt-core 1.8, I had the same issue on Python 3.9.6 (OS: Mac Ventura 13.4.1), and it was resolved on upgrade to Python 3.12.3.

[moseleyi]

I don't use venv nor poetry. At first I just upgraded pip install dbt-core --upgrade and the adapters. That didn't work so I tried uninstalling everything and installing again using different commands (core + adapters in one command, adapter only etc.)

All of them produce this error

[troyharvey]

Reproduced with Python 3.10.

＄ python --version
Python 3.10.14

＄ dbt deps
Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniconda/base/envs/ds-redshift/bin/dbt", line 5, in <module>
    from dbt.cli.main import cli
  File "/opt/homebrew/Caskroom/miniconda/base/envs/ds-redshift/lib/python3.10/site-packages/dbt/cli/__init__.py", line 1, in <module>
    from .main import cli as dbt_cli  # noqa
  File "/opt/homebrew/Caskroom/miniconda/base/envs/ds-redshift/lib/python3.10/site-packages/dbt/cli/main.py", line 14, in <module>
    from dbt.cli import requires, params as p
  File "/opt/homebrew/Caskroom/miniconda/base/envs/ds-redshift/lib/python3.10/site-packages/dbt/cli/requires.py", line 10, in <module>
    from dbt.adapters.factory import adapter_management, register_adapter, get_adapter
ModuleNotFoundError: No module named 'dbt.adapters.factory'

[paulbakkerbloom]

Had the same issue with Python 3.9.18 -> solved when I upgraded to Python 3.11.6

I use poetry and upgraded through poetry update

dbt-adapters                          1.1.1             The set of adapter ...
dbt-bigquery                          1.8.0             The Bigquery adapte...
dbt-common                            1.0.4             The shared common u...
dbt-core                              1.8.0             With dbt, data anal...
dbt-extractor                         0.5.1             A tool to analyze a...
dbt-semantic-interfaces               0.5.1             The shared semantic...
dbt-tests-adapter                     1.8.0             The set of reusable...

[moseleyi]

I was hoping it was about python issue but seems like it's not. It could be a dependency that should be updated with dbt core update but isn't being forced to update?

[moseleyi]

I also found this: AltimateAI/vscode-dbt-power-user#1029 but we all have dbt-adapters installed

[brabster]

A bit more info: I had the same problem, I can reproduce at will, Python 3.11.8:

Install dbt-bigquery==1.7.7 into a clean venv, working fine (via pip install -U -r requirements.txt)

Bump to dbt-bigquery>=1.7.7 and same pip command, broken

 File "/workspaces/pypi_vulnerabilities/venv/lib/python3.11/site-packages/dbt/cli/requires.py", line 10, in <module>
    from dbt.adapters.factory import adapter_management, register_adapter, get_adapter
ModuleNotFoundError: No module named 'dbt.adapters.factory'

Delete venv, same requirements.txt, pip install - get 1.8, works fine.

In the broken venv, the factory.py (and other) files in dbt/adapters is missing.

You should be able to reproduce these steps exactly with a codespace off https://github.com/brabster/pypi_vulnerabilities

[dbeatty10]

Thanks for this info @brabster 🏅

Could you share the contents of your requirements.txt file also?

[brabster]

@dbeatty10
It's just the dbt-bigquery dependency. Originally as per repo. https://github.com/brabster/pypi_vulnerabilities/blob/main/requirements.txt

Clean venv, set to dbt-bigquery==1.7.7 to force pre-1.8.0 and install. Works.
Then set to dbt-bigquery>=1.7.7 to upgrade to latest. Broken
Clean venv, install same requirements. Works

[jtcohen6]

@brabster If in step (2), you run pip install --upgrade dbt-core dbt-common dbt-adapters dbt-bigquery, do you still see the error?

(I'm hoping this might be because the underlying dependencies are not being upgraded, when they should be, but I'm not 100% positive: #10123)

[dbeatty10]

Thank you @brabster ! 🏅 I was able to reproduce the error using your description. See "reprex" below for full details.

@jtcohen6 when I tried running pip install --upgrade dbt-core dbt-common dbt-adapters dbt-bigquery, it still gave the same error.

Reprex

Create requirements files for both 1.7 and 1.8:

cat << EOF > requirements_1.7.txt
dbt-bigquery==1.7.7
EOF
cat << EOF > requirements_1.8.txt
dbt-adapters==1.1.1
dbt-common==1.0.4
dbt-core==1.8.0
dbt-extractor==0.5.1
dbt-bigquery==1.8.0
dbt-semantic-interfaces==0.5.1
EOF

✅ Create a fresh virtual environment and install the requirements for 1.7:

rm -rf env
python3 -m venv env
source env/bin/activate
python -m pip install -r requirements_1.7.txt
source env/bin/activate
dbt --version

❌ Try upgrading by installing the requirements for 1.8 in the same virtual environment:

python -m pip install -r requirements_1.8.txt
source env/bin/activate
dbt --version

Get an error like the following:

    from dbt.adapters.factory import adapter_management, register_adapter, get_adapter
ModuleNotFoundError: No module named 'dbt.adapters.factory'

✅ Try upgrading by installing the requirements for 1.8 in a fresh virtual environment:

deactivate
rm -rf env
python3 -m venv env
source env/bin/activate
python -m pip install -r requirements_1.8.txt
source env/bin/activate
dbt --version

[dbeatty10]

When I got the same error as the reported above, the following workaround resolved it for me:

python -m pip uninstall -y dbt-adapters
python -m pip install --upgrade dbt-core dbt-common dbt-adapters
dbt --version

The pip uninstall -y dbt-adapters is the important part.

The 2nd command can be replaced with whichever command you use to install the requirements for dbt 1.8.

[moseleyi]

Yes, this workaround seems to work. But what is the underlying issue?

[brabster]

Seems to be related to the move of dbt-adapters from part of dbt-core to standalone. I can upgrade no problem from a 1.8 pre-release to latest, but anything from before dbt-adapters being standalone and boom.

I can see in verbose pip install logs that it's going around deleting the files in dbt/adapters/... but no evidence that it knows to replace them. Weird. I wonder if it's something to do with going to a standalone dependency but having all the files end up in the same place, maybe a kind of race condition so they are already there when it's checking what to install, but then it goes and deletes them afterwards leaving you broken. Perhaps moving the standalone adapters directory tree away from where it used to be could resolve the problem? Not had chance to check

[dbeatty10]

@moseleyi I don't know the underlying root cause.

But I'm able to see some differences when I create two different virtual environments:

• env1 is the environment that has the error
• env2 is the environment that is working as expected

Difference 1

One of the differences is in this file: lib/python3.12/site-packages/dbt/__init__.py

• env1 has contents that are identical to dbt-postgres dbt/__init__.py
• env2 has contents that are identical to dbt-adapters dbt/__init__.py
• Neither environment has contents identical to dbt-core core/dbt/__init__.py

Difference 2

env2 has the following files/folders within lib/python3.12/site-packages/dbt/adapters that are not in env1:

• __init__.py
• cache.py
• capability.py
• factory.py
• protocol.py
• reference_keys.py
• sql

These are the files/folders that are within both:

• __about__.py
• __pycache__
• base
• clients
• contracts
• events
• exceptions
• postgres
• py.typed
• relation_configs
• utils.py

[dbeatty10]

Here's another command that resolved the error for me:

python -m pip install --force-reinstall dbt-adapters

[brabster]

For info pip install -U --force-reinstall -r requirements.txt does not work, unfortunately.

[brabster]

Following previous comments - I think it might be a bug in pip? Seems like the last thing pip does is uninstall the old versions including the old dbt-core, which used to contain the adapters directory and doesn't anymore. If it's deleting the common files with the new dbt-adapters package when it does that uninstall, that might explain the missing files when you try to upgrade from core to standalone adapters

Installing collected packages: daff, dbt-semantic-interfaces, dbt-common, dbt-adapters, dbt-core, dbt-bigquery
  Attempting uninstall: dbt-semantic-interfaces
    Found existing installation: dbt-semantic-interfaces 0.4.4
    Uninstalling dbt-semantic-interfaces-0.4.4:
      Successfully uninstalled dbt-semantic-interfaces-0.4.4
  Attempting uninstall: dbt-core
    Found existing installation: dbt-core 1.7.14
    Uninstalling dbt-core-1.7.14:
      Successfully uninstalled dbt-core-1.7.14
  Attempting uninstall: dbt-bigquery
    Found existing installation: dbt-bigquery 1.7.7
    Uninstalling dbt-bigquery-1.7.7:
      Successfully uninstalled dbt-bigquery-1.7.7
Successfully installed daff-1.3.46 dbt-adapters-1.1.1 dbt-bigquery-1.8.1 dbt-common-1.0.4 dbt-core-1.8.0 dbt-semantic-interfaces-0.5.1

[jkleinkauff]

Same problem here. @dbeatty10 's solution worked for me.

python -m pip uninstall -y dbt-adapters
python -m pip install --upgrade dbt-core dbt-common dbt-adapters
dbt --version

[brabster]

I had a look around in the pip source code and if I'm understanding this correctly, it's clear what the problem is and any upgrade from pre-standalone adapters to standalone adapters will break in this way.

After interrogating PyPI for the latest versions of things, we get this (verbose) log line like this:

Installing collected packages: daff, dbt-semantic-interfaces, dbt-common, dbt-adapters, dbt-core, dbt-bigquery

That's from line 54 here which enumerates to_install - an OrderedDict, so the packages are always in alpha order, and dbt-adapters comes before dbt-core (demonstrated in the debug line I showed).

Now on line 61, pip loops over the ordered requirements, i.e. dbt-adapters first and then dbt-core, and for each package uninstalls the previous version and installs the new version.

Because dbt-core follows dbt-adapters, the files they have in common get deleted as part of the dbt-core uninstall and leaves a broken installation.

It's certainly possible I'm wrong, but I attach a verbose upgrade log and you can see the ordering and uninstall activity happening in a manner consistent with this explanation towards the end.

pip.log

It also explains every reported behaviour in this issue:

• any attempt to update dbt-core in place with an existing version of dbt-core that included the adapter file structure will cause this break
• cleaning the venv first will avoid the reinstall behaviour and be successful
• --force-reinstall flag will make pip copy dbt-adapters back into place, fixing the break

If I can find an old version of dbt-core that didn't have those adapters files in that specific location, then I think an upgrade should work, if my theory is correct. Will let you know if I can verify that.

===== Update

Seems that adapters structure has been there since the first version of dbt-core in PyPI. Instead, I installed dbt-core==1.7.0 into a fresh venv, then hacked the site-packages/dbt-core...dist-info/RECORD file to remove all mention of anything in dbt/adapters. Then I pip install --upgrade to latest and the adapter files are there. It still breaks, but you can see the factory module is no longer causing the problem, all seems to suggest the theory is correct.

Installing collected packages: daff, sqlparse, protobuf, dbt-semantic-interfaces, dbt-common, dbt-adapters, dbt-core
  Attempting uninstall: sqlparse
    Found existing installation: sqlparse 0.4.4
    Uninstalling sqlparse-0.4.4:
      Successfully uninstalled sqlparse-0.4.4
  Attempting uninstall: protobuf
    Found existing installation: protobuf 5.26.1
    Uninstalling protobuf-5.26.1:
      Successfully uninstalled protobuf-5.26.1
  Attempting uninstall: dbt-semantic-interfaces
    Found existing installation: dbt-semantic-interfaces 0.4.4
    Uninstalling dbt-semantic-interfaces-0.4.4:
      Successfully uninstalled dbt-semantic-interfaces-0.4.4
  Attempting uninstall: dbt-core
    Found existing installation: dbt-core 1.7.0
    Uninstalling dbt-core-1.7.0:
      Successfully uninstalled dbt-core-1.7.0
Successfully installed daff-1.3.46 dbt-adapters-1.1.1 dbt-common-1.0.4 dbt-core-1.8.0 dbt-semantic-interfaces-0.5.1 protobuf-4.25.3 sqlparse-0.5.0
(venv) (venv) @brabster ➜ /workspaces/pypi_vulnerabilities (main) $ dbt debug
Traceback (most recent call last):
  File "/workspaces/pypi_vulnerabilities/venv/bin/dbt", line 5, in <module>
    from dbt.cli.main import cli
  File "/workspaces/pypi_vulnerabilities/venv/lib/python3.11/site-packages/dbt/cli/__init__.py", line 1, in <module>
    from .main import cli as dbt_cli  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/pypi_vulnerabilities/venv/lib/python3.11/site-packages/dbt/cli/main.py", line 14, in <module>
    from dbt.cli import requires, params as p
  File "/workspaces/pypi_vulnerabilities/venv/lib/python3.11/site-packages/dbt/cli/requires.py", line 10, in <module>
    from dbt.adapters.factory import adapter_management, register_adapter, get_adapter
  File "/workspaces/pypi_vulnerabilities/venv/lib/python3.11/site-packages/dbt/adapters/factory.py", line 21, in <module>
    from dbt.include.global_project import (
ImportError: cannot import name 'PACKAGE_PATH' from 'dbt.include.global_project' (unknown location)

Not sure how come dbt-adapters ends up in a common directory structure as dbt-core but strikes me as not ideal as it means it's not really independent. Any oversight and chance collision could cause weird problems on update. Setting dbt-adapters to install where I'd have expected it in its own director under site packages should solve the problem and avoid any regression?

[naveenwashere]

pip install --force-reinstall dbt-adapters

This worked for me. Thanks @dbeatty10 .

[karenderer]

I found a way to upgrade to 1.8 with poetry by downgrading dbt-adapters to 1.1.0 and then updating it to 1.1.1 but it's not ideal.

Details

$ poetry add --group dev dbt-core=^1.8.1 dbt-snowflake=^1.8.2
$ dbt --version

ModuleNotFoundError: No module named 'dbt.adapters.factory'

$ poetry add --group dev dbt-adapters = "^1.1.0"

change the following in pyproject.toml

• set dbt-adapters to 1.1.0
• set dbt-core to 1.8.0
• set dbt-snowflake to 1.8.0

$ poetry lock --no-update
$ poetry install --sync --no-root

$ dbt version

works

$ dbt debug

TypeError: __post_serialize__() got an unexpected keyword argument 'context'

change the following in pyproject.toml

• set dbt-adapters to 1.1.1

$ poetry lock --no-update
$ poetry install --sync --no-root

$ dbt version

works

$ dbt debug

runs successfully

[MikeMccree]

pip install --force-reinstall dbt-adapters

This worked for me. Thanks @dbeatty10 .

+1 worked for me as well.

I am on mac, ran the following:
pip install --upgrade \ dbt-core \ dbt-postgres \ dbt-redshift \ dbt-spark\ dbt-athena-community\ dbt-glue

Encountered the error with adapters. ran the force-reinstall and we all good.. Thanks @dbeatty10

[rizahmeds]

Upgraded python version from 3.11.5 to 3.12.3, and it worked.

[dun-x]

Here's another command that resolved the error for me:

python -m pip install --force-reinstall dbt-adapters

This work for me

[elyobo]

For those using poetry, my fix was to remove the env and reinstall, e.g.

poetry env remove 3.12
poetry install

[nicolefeatherby]

Ran into this too using a venv on python 3.12.0

pip install --force-reinstall dbt-adapters worked for me too

Got this message while doing so, but everything works, so not sure if relevant

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
snowflake-connector-python 3.5.0 requires urllib3<2.0.0,>=1.21.1, but you have urllib3 2.2.1 which is incompatible.