[WIP] Interoperability updates for py3.8, py3.9 and py3.10 #6157

nzlosh · 2024-02-28T13:36:32Z

This branch started as an exploration for upgrading mongoengine/pymongo but has evolved into aligning module dependencies and fixing code related to updating module versions to support py3.8, py3.9 and py3.10.

nzlosh · 2024-03-12T10:12:54Z

Currently attempting to get tests passing with pymongo 3.13 to display deprecation warns for python3.8 to python3.10.
nose is blocking python3.10 (Will attempt to run tests with pytest)
oslo.config is blocking python3.10 (Upgraded to latest, but behavioural changes break string output. Further investigation required.)

…' instead

… importlib

…lo.config

This should support the rest of the updated requirements copied from the lockfile.

cognifloyd · 2024-04-10T23:43:41Z

It looks like the next chunk in your PR that could be extracted into a new PR is replacing unittest2 with unittest.

cognifloyd · 2024-04-19T08:12:12Z

I cherry picked your unittest2 changes to #6187.

cognifloyd · 2024-04-19T18:15:06Z

Cherry-picked PRs:

Fix misc DeprecationWarnings #6188:
- Fix for DeprecationWarning: the imp module is deprecated in favour of importlib
  - 21ebac7
- Fixed more DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
  - ca7db1d
  - a87373c (w/o the pytest bits)
  - be108b3
- Fix for DeprecationWarning: This method will be removed in Python 3.12. Use 'parser.read_file()' instead.
  - c574b29
- Bump semver for py3.10 and fix DeprecationWarning: Function semver.match is deprecated and other semver warnings
  - bcc74d8
  - 6aaaaaf (just the semver bits)
Update st2client deps: editor and prompt-toolkit #6189
- Switch python-editor to editor
  - d8657cc
  - 9913406
- Use pygment token in place of prompt_toolkit
  - 5dcc31c
  - 758eb65
  - c5b1943
  - 45c256a
  - 1659e96 (just the pygments bits)
  - 9174603 (just the pygments bits)
Update pylint to 3.1.0 #6191
- 02b7beb (just the pylint bits)
- ddf4fbb
- de424bd
- 2c15886
- 12b9eda
- d18fb02 (partial)
- a6a0bfd
- a804694 (w/o copyright change)
- bdfbd77
- 15a2b6e
- 4668ed5
- 647fca8
Update oslo.config to prepare for python 3.10 support. #6193
- 1ad6c34 (only oslo.config bump in fixed-requirements.txt etc)
- 7d3990e
- a87373c (only msg change in st2api/tests/unit/test_validation_utils.py)
- 6b55018 (only msg change in st2api/tests/unit/test_validation_utils.py)
- c2d9d01 (only comment in fixed-requirements.txt)

Next PR(s) to extract:

Fix bash syntax error in test_pause_resume_with_error
- 33af575
❓ Removed test for application_headers when no compression enabled. and doc comment fix
- 2368557
stub test files for pytest
- 2998a0d (only contrib/runners/{local,remote}_runner/tests/unit/*)
- 02b7beb (only contrib/ changes)

Probably keep in the py3.10 specific PR:

change interpreter constraints to include py3.10 and regen lockfiles
py3.10 message differences
- 999aa4f
- 02b7beb (only fmt in st2client/tests/unit/test_commands.py)

st2client/tests/unit/test_shell.py

cognifloyd · 2024-05-02T15:10:59Z

pants.toml

+  # pants plugins - 202404 disabled because pants doesn't support >py3.9 and conflicts with other targets using >py3.9
+#  "/pants-plugins",


What issue(s) led to commenting this?

I believe this is what we need to change to add py3.10 to the interpreter constraints:
753974d

(That commit is in the pants-ic-py3.10 branch which includes the update-oslo.config branch)

On ubuntu 22.04 there is no python3.9 interpreter and no matter if I tried py3.10 or py3.11 or bump pants to 2.18 it refused to run because it claimed to need py3.9. I couldn't figure out how to find a version combination that appeased the constraints so I commented to be able to move on the st2 code base rather than battle with pants plugins. I eventually installed python3.9 from source on u2204 to be able to run pants 2.17.

cognifloyd · 2024-05-02T15:14:50Z

pants.toml

@@ -111,38 +111,36 @@ enable_resolves = true
 default_resolve = "st2"
 interpreter_constraints = [
  # python_distributions needs a single constraint (vs one line per python version).
-  "CPython>=3.8,<3.10",
+  "CPython==3.9.*",


This is the interpreter constraints for our code, so your change tells pants that st2 only supports py 3.9.

Suggested change

"CPython==3.9.*",

"CPython>=3.8,<3.11",

That probably explains why I couldn't get pants to run without py3.9

cognifloyd · 2024-05-02T15:16:11Z

pants-plugins/BUILD

@@ -2,7 +2,7 @@ __defaults__(
    all=dict(
        resolve="pants-plugins",
        skip_pylint=True,
-        interpreter_constraints=["CPython==3.9.*"],
+        interpreter_constraints=["CPython<=3.11,>=3.8"],


This code only runs in pants, so it should match pants constraints.

Suggested change

interpreter_constraints=["CPython<=3.11,>=3.8"],

interpreter_constraints=["CPython==3.9.*"],

BTW Pants will jump to python 3.11 in a future version. Once we update up that version, we can adjust this constraint.

I had a hard time with pants, so take all those settings with that in mind. :insert dog typing on keyboard meme here:

Note that this only adjusts interpreter constraints. python 3.10 is not officially supported yet. Lockfile diff: lockfiles/st2.lock [st2] == Upgraded dependencies == orjson 3.10.1 --> 3.10.2 Lockfile diff: lockfiles/bandit.lock [bandit] == Upgraded dependencies == setuptools 69.2.0 --> 69.5.1 Lockfile diff: lockfiles/flake8.lock [flake8] == Upgraded dependencies == setuptools 69.2.0 --> 69.5.1 Lockfile diff: lockfiles/black.lock [black] == Upgraded dependencies == platformdirs 4.2.0 --> 4.2.1 typing-extensions 4.10.0 --> 4.11.0 Lockfile diff: lockfiles/pytest.lock [pytest] == Upgraded dependencies == coverage 7.4.4 --> 7.5.0 execnet 2.0.2 --> 2.1.1 pluggy 1.4.0 --> 1.5.0 Lockfile diff: lockfiles/twine.lock [twine] == Upgraded dependencies == idna 3.6 --> 3.7 jaraco-context 4.3.0 --> 5.3.0 jaraco-functools 4.0.0 --> 4.0.1 keyring 25.1.0 --> 25.2.0 == Added dependencies == backports-tarfile 1.1.1 Lockfile diff: lockfiles/pants-plugins.lock [pants-plugins] == Upgraded dependencies == pluggy 1.4.0 --> 1.5.0

Lockfile diff: lockfiles/st2.lock [st2] == Upgraded dependencies == bcrypt 4.1.2 --> 4.1.3 chardet 3.0.4 --> 5.2.0 cryptography 42.0.5 --> 42.0.7 dnspython 1.16.0 --> 2.6.1 flask 2.1.3 --> 3.0.3 jinja2 3.1.3 --> 3.1.4 markupsafe 2.0.1 --> 2.1.5 networkx 2.8.8 --> 3.1 orjson 3.10.2 --> 3.10.3 platformdirs 4.2.1 --> 4.2.2 pygments 2.17.2 --> 2.18.0 tenacity 8.2.3 --> 8.3.0 ujson 5.9.0 --> 5.10.0 virtualenv 20.26.1 --> 20.26.2 werkzeug 2.1.2 --> 3.0.3 zipp 3.18.1 --> 3.18.2 == Added dependencies == blinker 1.8.2 == Removed dependencies == colorama 0.4.6 nose 1.3.7 nose-parallel 0.4.0 nose-timer 1.0.1 rednose 1.3.0 termstyle 0.1.11

Apparently, argparse has been in the standard library since Python 3.2. We want the std lib version, not the old version from pypi. So, drop the argparse dep.

cognifloyd · 2024-05-17T03:53:36Z

@nzlosh I resolved the merge conflicts, fixed the interpreter constraints, and updated the lockfiles. Hopefully this provides a good basis for you to keep keep working on py 3.10 support. I'm not planning to do much else with requirements, so I hope my other pants work won't create [too many] merge conflicts for you.

nzlosh · 2024-05-17T06:59:14Z

@cognifloyd You're a legend!! Thanks for sorting that mess out. I will create smaller PRs against master as I progress when I pickup the 3.10 compatibility work. 17 out of 29 checks passing is encouraging.

pull-request-size bot added the size/L PR that changes 100-499 lines. Requires some effort to review. label Feb 28, 2024

nzlosh marked this pull request as draft February 28, 2024 13:37

nzlosh force-pushed the mongo_upgrade branch from 16e399c to cd769fd Compare February 28, 2024 13:52

pull-request-size bot added size/XS PR that changes 0-9 lines. Quick fix/merge. and removed size/L PR that changes 100-499 lines. Requires some effort to review. labels Feb 28, 2024

Update pymongo and mongoengine to display deprecation warnings.

21e34fb

nzlosh force-pushed the mongo_upgrade branch from cd769fd to 21e34fb Compare February 28, 2024 14:15

pull-request-size bot added size/S PR that changes 10-29 lines. Very easy to review. size/M PR that changes 30-99 lines. Good size to review. and removed size/XS PR that changes 0-9 lines. Quick fix/merge. size/S PR that changes 10-29 lines. Very easy to review. labels Feb 28, 2024

Bump version for py3.10 support

1ad6c34

nzlosh force-pushed the mongo_upgrade branch from fcd6d67 to 1ad6c34 Compare March 6, 2024 13:41

nzlosh mentioned this pull request Mar 12, 2024

TSC Meeting (12 March 2024) StackStorm/community#133

Open

nzlosh added 2 commits March 12, 2024 13:24

Replace nose with pytest

634a7b3

Remove unittest2, switch nosetest to pytest

2102851

pull-request-size bot added size/XL PR that changes 500-999 lines. Consider splitting work into several ones that easier to review. and removed size/M PR that changes 30-99 lines. Good size to review. labels Mar 12, 2024

nzlosh added 7 commits March 12, 2024 14:35

update requirements

12b8254

Replace nosetest with pytest (without concurrency)

071c2b0

Fix for DeprecationWarning: Please use assertRaisesRegex

a1d35a7

Fix DeprecationWarning: The 'warn' method is deprecated, use 'warning…

ca7db1d

…' instead

Fix DeprecationWarning: Please use assertRegex instead.

4cbbdfc

Fix for DeprecationWarning: Please use assertNotEqual instead.

848d9c2

Fix for DeprecationWarning: the imp module is deprecated in favour of…

21ebac7

… importlib

nzlosh force-pushed the mongo_upgrade branch from 52b513e to 21ebac7 Compare March 13, 2024 13:45

Fix ssl_cert_reqs choices and update data type of newer version of os…

7d3990e

…lo.config

nzlosh force-pushed the mongo_upgrade branch from 582e68a to 7d3990e Compare March 13, 2024 14:39

Enable testing with py3.10

9753982

cognifloyd added a commit that referenced this pull request Apr 10, 2024

Copy rstcheck update from @nzlosh in #6157

d039a4d

This should support the rest of the updated requirements copied from the lockfile.

cognifloyd added a commit that referenced this pull request Apr 10, 2024

Copy rstcheck update from @nzlosh in #6157

9d0db9f

This should support the rest of the updated requirements copied from the lockfile.

cognifloyd added a commit that referenced this pull request Apr 10, 2024

Copy pip-tools update from @nzlosh in #6157

d7c8df3

This should support the rest of the updated requirements copied from the lockfile.

cognifloyd mentioned this pull request Apr 12, 2024

Use pants managed venv in Makefile #6130

Draft

cognifloyd mentioned this pull request Apr 19, 2024

Drop unittest2 #6187

Merged

cognifloyd mentioned this pull request Apr 19, 2024

Refactor tools/launchdev.sh to use tmux #6186

Merged

This was referenced Apr 24, 2024

Fix misc DeprecationWarnings #6188

Merged

Update st2client deps: editor and prompt-toolkit #6189

Merged

cognifloyd reviewed Apr 24, 2024

View reviewed changes

st2client/tests/unit/test_shell.py Show resolved Hide resolved

This was referenced Apr 25, 2024

Update pylint to 3.1.0 #6191

Merged

Update oslo.config to prepare for python 3.10 support. #6193

Merged

cognifloyd reviewed May 2, 2024

View reviewed changes

winem mentioned this pull request May 14, 2024

TSC Meeting (14 May 2024) StackStorm/community#137

Open

Merge branch 'master' into mongo_upgrade

7b17a85

pull-request-size bot added size/L PR that changes 100-499 lines. Requires some effort to review. and removed size/XXL PR that changes 1000+ lines. You should absolutely split your PR into several. labels May 16, 2024

pull-request-size bot added size/XXL PR that changes 1000+ lines. You should absolutely split your PR into several. and removed size/L PR that changes 100-499 lines. Requires some effort to review. labels May 16, 2024

cognifloyd added 6 commits May 16, 2024 21:16

loosen some fixed requirements

3167134

revert some pants changes

ec74e43

remove some constraints from lockfiles/st2-constraints.txt

eb9d408

update fixed-requirements.txt using versions from lockfiles/st2.lock

bd629b4

Remove std lib dep (argparse) from requirements

aba09f3

Apparently, argparse has been in the standard library since Python 3.2. We want the std lib version, not the old version from pypi. So, drop the argparse dep.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Interoperability updates for py3.8, py3.9 and py3.10 #6157

[WIP] Interoperability updates for py3.8, py3.9 and py3.10 #6157

nzlosh commented Feb 28, 2024 •

edited

nzlosh commented Mar 12, 2024 •

edited

cognifloyd commented Apr 10, 2024

cognifloyd commented Apr 19, 2024

cognifloyd commented Apr 19, 2024 •

edited

cognifloyd May 2, 2024

nzlosh May 2, 2024

cognifloyd May 2, 2024 •

edited

nzlosh May 2, 2024 •

edited

cognifloyd May 2, 2024

nzlosh May 2, 2024

cognifloyd commented May 17, 2024 •

edited

nzlosh commented May 17, 2024

		# pants plugins - 202404 disabled because pants doesn't support >py3.9 and conflicts with other targets using >py3.9
		# "/pants-plugins",

	interpreter_constraints=["CPython<=3.11,>=3.8"],
	interpreter_constraints=["CPython==3.9.*"],

[WIP] Interoperability updates for py3.8, py3.9 and py3.10 #6157

Are you sure you want to change the base?

[WIP] Interoperability updates for py3.8, py3.9 and py3.10 #6157

Conversation

nzlosh commented Feb 28, 2024 • edited

nzlosh commented Mar 12, 2024 • edited

cognifloyd commented Apr 10, 2024

cognifloyd commented Apr 19, 2024

cognifloyd commented Apr 19, 2024 • edited

cognifloyd May 2, 2024

Choose a reason for hiding this comment

nzlosh May 2, 2024

Choose a reason for hiding this comment

cognifloyd May 2, 2024 • edited

Choose a reason for hiding this comment

nzlosh May 2, 2024 • edited

Choose a reason for hiding this comment

cognifloyd May 2, 2024

Choose a reason for hiding this comment

nzlosh May 2, 2024

Choose a reason for hiding this comment

cognifloyd commented May 17, 2024 • edited

nzlosh commented May 17, 2024

nzlosh commented Feb 28, 2024 •

edited

nzlosh commented Mar 12, 2024 •

edited

cognifloyd commented Apr 19, 2024 •

edited

cognifloyd May 2, 2024 •

edited

nzlosh May 2, 2024 •

edited

cognifloyd commented May 17, 2024 •

edited