Code changes for SQLAlchemy 2.0.x upgrade

[aanand-gsa]

This PR proposes the solution for the breaking changes introduced on SQLAlchemy 2.0 upgrade https://docs.sqlalchemy.org/en/20/changelog/migration_20.html

Details

This changeset addresses all of the changes we need to make to our app and tests in order to upgrade to the 2.0.x series of SQLAlchemy. It builds upon the work we previously completed to upgrade Python to 3.12.x and Flask 3.0.x, and accounts for the remaining dependency updates that are shared between Flask and SQLAlchemy.

Security Considerations

• Updates SQLAlchemy to the 2.0.x series so that we can continue to benefit from new releases and security patches.
• Updates several Flask and SQLAlchemy plugins as well so that those also remain up-to-date.
• Let's us take advantage of the newer updates that Python, Flask, and SQLAlchemy all have to offer us.

[ccostino]

A quick note to folks reviewing this PR: several other dependency updates have been merged in from main over the last couple of days as of now, so you may need to do a pull on this branch and run make bootstrap again to make sure you have all of the latest things.

This was done to keep the PR fresh and in sync with main to make any other updates and eventual merge straightforward.

[xlorepdarkhelm]

Attempted "Send a message from an existing template", the following error happened:

09:38:37 worker.1    | [2024-05-08 09:38:37,059: INFO/ForkPoolWorker-2] Start sending SMS for notification id: a110a6fb-bac3-4a06-805e-48f63bac14fb
09:38:37 web.1       | 2024-05-08T09:38:37 app werkzeug INFO no-request-id no-service-id "127.0.0.1 - - [08/May/2024 09:38:37] "GET /service/bfac885a-5de4-4f3b-8f69-fa8518617e3a/notification-count HTTP/1.1" 200 -" [in /Users/chill/.virtualenvs/Notify.gov-api/lib/python3.12/site-packages/werkzeug/_internal.py:97]
09:38:37 worker.1    | 2024-05-08T09:38:37 delivery delivery INFO no-request-id no-service-id "Duplicate callback received for service bfac885a-5de4-4f3b-8f69-fa8518617e3a. Notification ID a110a6fb-bac3-4a06-805e-48f63bac14fb with type sms sent by None. New status was temporary-failure, current status is temporary-failure. This happened 0:00:00.699610 after being first set." [in /Users/chill/Projects/Flexion/GSA/Notify.gov/notifications-api/app/dao/notifications_dao.py:615]
09:38:37 worker.1    | [2024-05-08 09:38:37,742: INFO/ForkPoolWorker-2] Duplicate callback received for service bfac885a-5de4-4f3b-8f69-fa8518617e3a. Notification ID a110a6fb-bac3-4a06-805e-48f63bac14fb with type sms sent by None. New status was temporary-failure, current status is temporary-failure. This happened 0:00:00.699610 after being first set.
09:38:37 worker.1    | 2024-05-08T09:38:37 delivery delivery ERROR no-request-id no-service-id "SMS notification delivery for id: a110a6fb-bac3-4a06-805e-48f63bac14fb failed" [in /Users/chill/Projects/Flexion/GSA/Notify.gov/notifications-api/app/celery/provider_tasks.py:135]
09:38:37 worker.1    | Traceback (most recent call last):
09:38:37 worker.1    |   File "/Users/chill/Projects/Flexion/GSA/Notify.gov/notifications-api/app/celery/provider_tasks.py", line 113, in deliver_sms
09:38:37 worker.1    |     message_id = send_to_providers.send_sms_to_provider(notification)
09:38:37 worker.1    |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
09:38:37 worker.1    |   File "/Users/chill/Projects/Flexion/GSA/Notify.gov/notifications-api/app/delivery/send_to_providers.py", line 35, in send_sms_to_provider
09:38:37 worker.1    |     personalisation = get_personalisation_from_s3(
09:38:37 worker.1    |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
09:38:37 worker.1    |   File "/Users/chill/Projects/Flexion/GSA/Notify.gov/notifications-api/app/aws/s3.py", line 200, in get_personalisation_from_s3
09:38:37 worker.1    |     incr_jobs_cache_misses()
09:38:37 worker.1    |   File "/Users/chill/Projects/Flexion/GSA/Notify.gov/notifications-api/app/aws/s3.py", line 85, in incr_jobs_cache_misses
09:38:37 worker.1    |     hits = redis_store.get(JOBS_CACHE_HITS).decode("utf-8")
09:38:37 worker.1    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
09:38:37 worker.1    | AttributeError: 'NoneType' object has no attribute 'decode'
09:38:37 worker.1    | [2024-05-08 09:38:37,744: ERROR/ForkPoolWorker-2] SMS notification delivery for id: a110a6fb-bac3-4a06-805e-48f63bac14fb failed
09:38:37 worker.1    | Traceback (most recent call last):
09:38:37 worker.1    |   File "/Users/chill/Projects/Flexion/GSA/Notify.gov/notifications-api/app/celery/provider_tasks.py", line 113, in deliver_sms
09:38:37 worker.1    |     message_id = send_to_providers.send_sms_to_provider(notification)
09:38:37 worker.1    |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
09:38:37 worker.1    |   File "/Users/chill/Projects/Flexion/GSA/Notify.gov/notifications-api/app/delivery/send_to_providers.py", line 35, in send_sms_to_provider
09:38:37 worker.1    |     personalisation = get_personalisation_from_s3(
09:38:37 worker.1    |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
09:38:37 worker.1    |   File "/Users/chill/Projects/Flexion/GSA/Notify.gov/notifications-api/app/aws/s3.py", line 200, in get_personalisation_from_s3
09:38:37 worker.1    |     incr_jobs_cache_misses()
09:38:37 worker.1    |   File "/Users/chill/Projects/Flexion/GSA/Notify.gov/notifications-api/app/aws/s3.py", line 85, in incr_jobs_cache_misses
09:38:37 worker.1    |     hits = redis_store.get(JOBS_CACHE_HITS).decode("utf-8")
09:38:37 worker.1    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
09:38:37 worker.1    | AttributeError: 'NoneType' object has no attribute 'decode'

And I received this on the frontend:

Sorry, we can't deliver what you asked for right now.
Please try again later or email us for more information.

Did not receive a text.

This happens when I manually type in my phone number rather than pressing the "Use my number" button.

Otherwise, everything looks good. All tests run, all came back OK, other than what I noted above. Since it is involved with the sending process, it doesn't seem to matter if you are using an existing template, create a new one, or copy a template first. It does seem to be totally around the step where you can choose to manually enter the phone number.

[xlorepdarkhelm]

Noting this, as I found it during testing, but it seems completely unrelated to this change:

While checking the SQLAlchemy update, I found this error on bootup of the api, which might be important:

12:45:00 worker.1    | 2024-05-07T12:45:00 delivery delivery ERROR no-request-id no-service-id "RETRY: Email notification 4618446f-eab1-442d-b216-43cb17176233 failed" [in /Users/chill/Projects/Flexion/GSA/Notify.gov/notifications-api/app/celery/provider_tasks.py:186]
12:45:00 worker.1    | Traceback (most recent call last):
12:45:00 worker.1    |   File "/Users/chill/Projects/Flexion/GSA/Notify.gov/notifications-api/app/celery/provider_tasks.py", line 172, in deliver_email
12:45:00 worker.1    |     notification.personalisation = json.loads(personalisation)
12:45:00 worker.1    |                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
12:45:00 worker.1    |   File "/Users/chill/.pyenv/versions/3.12.2/lib/python3.12/json/__init__.py", line 339, in loads
12:45:00 worker.1    |     raise TypeError(f'the JSON object must be str, bytes or bytearray, '
12:45:00 worker.1    | TypeError: the JSON object must be str, bytes or bytearray, not NoneType
12:45:00 worker.1    | [2024-05-07 12:45:00,079: ERROR/ForkPoolWorker-2] RETRY: Email notification 4618446f-eab1-442d-b216-43cb17176233 failed
12:45:00 worker.1    | Traceback (most recent call last):
12:45:00 worker.1    |   File "/Users/chill/Projects/Flexion/GSA/Notify.gov/notifications-api/app/celery/provider_tasks.py", line 172, in deliver_email
12:45:00 worker.1    |     notification.personalisation = json.loads(personalisation)
12:45:00 worker.1    |                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
12:45:00 worker.1    |   File "/Users/chill/.pyenv/versions/3.12.2/lib/python3.12/json/__init__.py", line 339, in loads
12:45:00 worker.1    |     raise TypeError(f'the JSON object must be str, bytes or bytearray, '
12:45:00 worker.1    | TypeError: the JSON object must be str, bytes or bytearray, not NoneType
12:45:00 worker.1    | [2024-05-07 12:45:00,091: INFO/MainProcess] Task deliver_email[375a473b-d1f5-48f6-b363-0feb579fb36b] received
12:45:00 worker.1    | [2024-05-07 12:45:00,094: INFO/ForkPoolWorker-2] Task deliver_email[375a473b-d1f5-48f6-b363-0feb579fb36b] retry: Retry in 300s

[xlorepdarkhelm]

I am not able to duplicate the errors I saw now, so they exist, but I can't make them happen. Anyway, the system seems G2G to be merged, everything looks fine.

[ccostino]

The startup error I've not been able to duplicate either and is likely the result of a local dev environment setup thing; we won't really know for sure if it's an issue with startup until it's put into one of the other cloud.gov environments, but I'm not seeing any indication that it would be.

The personalization error when scheduling a send has already been flagged as a separate issue, so that's also accounted for and not related to this change.

It does look like we're good to go here to move forward!

[anagradova]

LGTM

[ccostino]

A big thank you to @aanand-gsa for all of the hard work making this migration to SQLAlchemy 2.0.x possible for Notify.gov! 🙂 🎉

[aanand-gsa]

A big thank you to @aanand-gsa for all of the hard work making this migration to SQLAlchemy 2.0.x possible for Notify.gov! 🙂 🎉

Thanks @ccostino for the appreciation! It's been a team effort, and I'm glad we could successfully migrate to SQLAlchemy 2.0.x for Notify.gov. 😊 🙌