Unexpected error on zappa update - ImproperlyConfigured: Error loading psycopg2 or psycopg module on AWS

[spirovskib]

Context

I was pushing a minor code update to a zappa managed Django site.
The site was operating normally until the update.
On the update I got the following error

Waiting for lambda function [goya-core-production] to be updated...
Error: Warning! Status check on the deployed lambda failed. A GET request to '/' yielded a 502 response code.

When running zappa tail production i found the following error:

[1690488128872] [ERROR] ImproperlyConfigured: Error loading psycopg2 or psycopg module
File "/var/lang/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/var/task/django/db/backends/postgresql/base.py", line 29, in <module>
    raise ImproperlyConfigured("Error loading psycopg2 or psycopg module")

During the preparation of the deployment i don't see that psycopg2-binary is loaded from manylinux wheel.
The only change i've made was updating to MacOS Ventura 13.5 yesterday. But I'm using homebrew python.

Expected Behavior

zappa update production should properly deploy the application and run it, since no library changes were made

Actual Behavior

Deployment results in an error ImproperlyConfigured: Error loading psycopg2 or psycopg module

Possible Fix

No idea

Steps to Reproduce

Can't be sure that this reproduces, but this is my setup:
Have a lambda django application configured to connect to postgress database backend
On MacOS Ventura 13.5 have a Homebrew Python 3.8 install, running in virtual environment for the code
try a zappa update of the running code.

Your Environment

• Zappa version used: zappa -v 0.57.0
• Operating System and Python version:

MacOS Ventura 13.5, Macbook M1 CPU

Python 3.8.17 (default, Jun 21 2023, 05:28:43)
[Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin

Homebrew Python install, running in virtual environment for this code

• The output of pip freeze:

alt-fake-useragent==0.2.1
argcomplete==2.0.0
asgiref==3.6.0
backports.zoneinfo==0.2.1
beautifulsoup4==4.12.2
bleach==5.0.0
blurhash==1.1.4
boto3==1.26.114
botocore==1.29.114
build==0.8.0
certifi==2022.12.7
cfn-flip==1.3.0
charset-normalizer==2.1.0
click==8.1.3
configparser==5.3.0
courlan==0.7.1
crispy-bootstrap5==0.7
dateparser==1.1.8
decorator==5.1.1
Django==4.2.3
django-bleach==3.0.1
django-bootstrap-v5==1.0.11
django-ckeditor==6.5.1
django-cleanup==7.0.0
django-countries==7.5.1
django-crispy-forms==2.0
django-js-asset==2.0.0
django-meta==2.2.0
django-ranged-response==0.2.0
django-simple-captcha==0.5.17
django-storages==1.13.2
django-taggit==3.1.0
djangorestframework==3.14.0
dnspython==2.3.0
docutils==0.19
durationpy==0.5
email-validator==2.0.0.post2
feedparser==6.0.10
flake8==6.0.0
fpdf==1.7.2
future==0.18.3
fuzzywuzzy==0.18.0
git-filter-repo==2.38.0
hjson==3.1.0
htmldate==1.3.0
idna==3.4
jarowinkler==1.2.3
jmespath==1.0.1
joblib==1.2.0
jusText==3.0.0
kappa==0.6.0
langcodes==3.3.0
Levenshtein==0.20.1
lxml==4.9.1
markdownify==0.11.6
MarkupSafe==2.1.2
Mastodon.py==1.8.1
mccabe==0.7.0
mixpanel==4.10.0
nltk==3.7
oauthlib==3.2.2
packaging==21.3
pep517==0.13.0
Pillow==9.5.0
pip-tools==6.13.0
placebo==0.9.0
psycopg2-binary==2.9.4
pycodestyle==2.10.0
pyflakes==3.0.1
pyparsing==3.0.9
python-dateutil==2.8.2
python-dotenv==1.0.0
python-magic==0.4.27
python-slugify==8.0.1
pytz==2023.3
pytz-deprecation-shim==0.1.0.post0
PyYAML==6.0.1
rapidfuzz==2.15.1
regex==2023.3.23
requests==2.28.2
requests-oauthlib==1.3.1
s3transfer==0.6.0
sgmllib3k==1.0.0
six==1.16.0
slack-sdk==3.21.1
smart-open==6.3.0
soupsieve==2.3.2.post1
sqlparse==0.4.4
text-unidecode==1.3
tinycss2==1.2.1
tld==0.13
toml==0.10.2
tomli==2.0.1
tqdm==4.64.0
troposphere==4.0.2
tweepy==4.14.0
typing_extensions==4.5.0
tzdata==2023.3
tzlocal==4.3
urllib3==1.26.15
webencodings==0.5.1
Werkzeug==2.2.3
wsgi-request-logger==0.4.6
zappa==0.57.0

• Link to your project (optional): closed source, but the site should be visible here https://www.beyondmachines.net/
• Your zappa_settings.json:

{
    "production": {
        "django_settings": "goya_core.settings",
        "aws_region": "eu-central-1",
        "profile_name": "_profile_name_redacted_",
        "project_name": "goya-core",
        "runtime": "python3.8",
        "s3_bucket": "goya-zappa-files",
        "exclude": ["local_data_store",".git","*.md","CODEOWNERS","github_key.txt", "secret_key.txt", "*.sqlite3", "posgtres_env.json","static_assets","media_assets","static_data","*.code-workspace","Dockerfile","docker-compose.yml","local_data_store","zappashell.sh"],
        "timeout_seconds": 600,
        "manage_roles": false,
        "role_name": "ZappaRuntimeRole",
        "vpc_config": {
            "SubnetIds": [ "subnet-0d390740cf414cb13", "subnet-0f6e2a7f4a66cfeeb" ], 
            "SecurityGroupIds": [ "sg-03656bdc887694ed8" ]
        },
        "xray_tracing": false,
        "certificate_arn": "arn:aws:acm:us-east-1:_account_redacted_:certificate/_cert_redacted",
        "domain": "*.beyondmachines.net",
        "debug": false,
        "log_level": "DEBUG",
        "events": [{
            "function": "communicator.views.social_media_poster",
            "expression": "cron(1 8-20 ? * MON-SUN *)"
        },
        {
            "function": "content.views.generate_weekly_report_content",
            "expression": "cron(0 16 ? * MON *)"
        },
        {
            "function": "content.views.get_daily_events",
            "expression": "cron(0 10 ? * MON-FRI *)"
        }
        ]
    }
}

[spirovskib]

Update, how I made it work after 4 hours:

1. deactivate virtualenv
2. remove the entire virtualenv (the bin and lib folders and pyenv.cfg)
3. recreate the virtalenv with python 3.9 in the same folder virtualenv -p python3.9 .
4. navigate to the folder with requirements.txt and install them without cache pip install -r requirements.txt --no-cache-dir
5. test whether Django will run locally python manage.py runserver
6. update the deployment zappa update production
   On this iteration the loading of libraries includes psycopg2-binary==2.9.4: Using locally cached manylinux wheel (quoting the second run of the update so there is cache):

Downloading and installing dependencies..
 - levenshtein==0.20.1: Downloading
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.43M/1.43M [00:00<00:00, 3.69MB/s]
 - markupsafe==2.1.2: Downloading
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 25.5k/25.5k [00:00<00:00, 3.82MB/s]
 - pillow==9.5.0: Downloading
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3.31M/3.31M [00:01<00:00, 3.29MB/s]
 - pyyaml==6.0.1: Downloading
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 739k/739k [00:00<00:00, 3.41MB/s]
 - jarowinkler==1.2.3: Using locally cached manylinux wheel
 - lxml==4.9.1: Using locally cached manylinux wheel
 - psycopg2-binary==2.9.4: Using locally cached manylinux wheel
 - rapidfuzz==2.15.1: Using locally cached manylinux wheel
 - regex==2023.3.23: Using locally cached manylinux wheel

[spirovskib]

I was reviewing my actions last evening, and although I've managed to recover, it wasn't because I discovered the root cause. I'm reopening the issue for wider consideration, maybe some extra debugging is possible for everyone to benefit if this event happens again.

[monkut]

I assume that this problem occurred due to the latest version of psycopg2-binary not having a manylinux wheel available for the desired python runtime.
If that's the case, I recommend pinning the version of psycopg2-binary to the latest version where a manylinux wheel is available.

[spirovskib]

This is a very good insight. Is it possible to add a check and fail the update command if the manylinux wheel is not available and architecture of the machine is not x64?

[monkut]

That's a good point.
It would be better to abort the update when the associated library is not able to be installed.

At the moment the lambda package is built on the system where the command is run.
I suspect there could be a check during the package build process to determine if a package has a wheel or not.
Although it might be hard to determine which libraries would need a 'wheel' (typically those that require to be compiled/built) and those that don't.

In addition, we do allow deployment to docker, so that would need to be taken into consideration. (Since a docker deployment could potentially support the case where the expected 'wheel' is not available.

[RadySonabu]

When I pin this packages, it works on my end.
psycopg==3.1.10
psycopg-binary==3.1.10
psycopg2-binary==2.9.7
zappa==0.57.0