Error when setting up a development environment: “error in suds-jurko setup command: use_2to3 is invalid.”

[quinncomendant]

Hello,

I'm trying to get lexicon set up for local development. I'm following the instructions here. When I run poetry install -E full it fails when trying to install suds-jurko with the error error in suds-jurko setup command: use_2to3 is invalid. (full logs below). I wonder if the developers can reproduce this error too, or is it an issue with my system?

I'm starting with a fresh clone of https://github.com/AnalogJ/lexicon. Poetry appears to be installing the virtualenv correctly (python and pip are executing out of .venv/bin/…. I'm on macOS 10.14.6, with Python 3.9.6, Poetry 1.1.10, and pip 21.2.4.

I've never used Poetry before, so I'm starting with a fresh install and config of it.

Here's the full output when trying to install:

#> poetry install -E full
Creating virtualenv dns-lexicon in /Users/quinn/src/lexicon-upstream/.venv
Installing dependencies from lock file

Package operations: 81 installs, 0 updates, 0 removals

  • Installing pycparser (2.20)
  • Installing pyparsing (2.4.7)
  • Installing six (1.16.0)
  • Installing attrs (21.2.0)
  • Installing certifi (2021.5.30)
  • Installing cffi (1.14.6)
  • Installing charset-normalizer (2.0.4)
  • Installing idna (3.2)
  • Installing iniconfig (1.1.1)
  • Installing jmespath (0.10.0)
  • Installing packaging (21.0)
  • Installing pluggy (0.13.1)
  • Installing py (1.10.0)
  • Installing python-dateutil (2.8.2)
  • Installing toml (0.10.2)
  • Installing urllib3 (1.26.6)
  • Installing backports.entry-points-selectable (1.1.0)
  • Installing botocore (1.21.17)
  • Installing cryptography (3.4.7)
  • Installing distlib (0.3.2)
  • Installing filelock (3.0.12)
  • Installing multidict (5.1.0)
  • Installing platformdirs (2.2.0)
  • Installing pytest (6.2.4)
  • Installing requests (2.26.0)
  • Installing wcwidth (0.2.5)
  • Installing appdirs (1.4.4)
  • Installing cached-property (1.5.2)
  • Installing click (8.0.1)
  • Installing configparser (4.0.2)
  • Installing coverage (5.5)
  • Installing defusedxml (0.7.1)
  • Installing dnspython (2.1.0)
  • Installing execnet (1.9.0)
  • Installing isodate (0.6.0)
  • Installing lxml (4.6.3)
  • Installing mccabe (0.6.1)
  • Installing mypy-extensions (0.4.3)
  • Installing pathspec (0.9.0)
  • Installing prompt-toolkit (3.0.3)
  • Installing ptable (0.9.2)
  • Installing pycodestyle (2.7.0)
  • Installing pyflakes (2.3.1)
  • Installing pygments (2.9.0)
  • Installing pyopenssl (19.1.0)
  • Installing pytest-forked (1.3.0)
  • Installing pytz (2021.1)
  • Installing pyyaml (5.4.1)
  • Installing regex (2021.8.3)
  • Installing requests-file (1.5.1)
  • Installing requests-toolbelt (0.9.1)
  • Installing s3transfer (0.5.0)
  • Installing soupsieve (2.2.1)
  • Installing suds-jurko (0.6): Failed

  EnvCommandError

  Command ['/Users/quinn/src/lexicon-upstream/.venv/bin/pip', 'install', '--no-deps', 'file:///Users/quinn/Library/Caches/pypoetry/artifacts/a1/3a/ca/4cb96176d922bcef591bfe698c333a43a7173cf8a665798cd3a2614ec5/suds-jurko-0.6.tar.bz2'] errored with the following return code 1, and output:
  Processing /Users/quinn/Library/Caches/pypoetry/artifacts/a1/3a/ca/4cb96176d922bcef591bfe698c333a43a7173cf8a665798cd3a2614ec5/suds-jurko-0.6.tar.bz2
      ERROR: Command errored out with exit status 1:
       command: /Users/quinn/src/lexicon-upstream/.venv/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/sp/tqvnvb415cz3h7s5y20nx9dm0000gn/T/pip-req-build-sho1ojhu/setup.py'"'"'; __file__='"'"'/private/var/folders/sp/tqvnvb415cz3h7s5y20nx9dm0000gn/T/pip-req-build-sho1ojhu/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/sp/tqvnvb415cz3h7s5y20nx9dm0000gn/T/pip-pip-egg-info-vp06u4lf
           cwd: /private/var/folders/sp/tqvnvb415cz3h7s5y20nx9dm0000gn/T/pip-req-build-sho1ojhu/
      Complete output (1 lines):
      error in suds-jurko setup command: use_2to3 is invalid.
      ----------------------------------------
  WARNING: Discarding file:///Users/quinn/Library/Caches/pypoetry/artifacts/a1/3a/ca/4cb96176d922bcef591bfe698c333a43a7173cf8a665798cd3a2614ec5/suds-jurko-0.6.tar.bz2. Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.


  at ~/.poetry/lib/poetry/utils/env.py:1180 in _run
      1176│                 output = subprocess.check_output(
      1177│                     cmd, stderr=subprocess.STDOUT, **kwargs
      1178│                 )
      1179│         except CalledProcessError as e:
    → 1180│             raise EnvCommandError(e, input=input_)
      1181│
      1182│         return decode(output)
      1183│
      1184│     def execute(self, bin, *args, **kwargs):

  • Installing typed-ast (1.4.3)
  • Installing typing-extensions (3.10.0.0)
  • Installing virtualenv (20.7.1)
  • Installing wrapt (1.12.1)
  • Installing yarl (1.6.3)

I tried installing suds-jurko manually with source .venv/bin/activate && pip install --no-deps suds-jurko but that gives me the same error.

Update 1: I found this comment which says the use_2to3 flag no longer works as of setuptools version 58.0.2. Perhaps it would work to set a fixed setuptools version of < 58 when installing it? I'm not sure how to do this, but it appears it is currently defined in pyproject.toml: types-setuptools = "*".

Update 2: Ok, I've confirmed the above. If I downgrade setuptools to v56.2.0, then the poetry install -E full installation completes successfully (suds-jurko installs without error). Also, I found issue #2784 for pypa/setuptools which has comments that suggest the problem is an update required in suds-jurko. The resolution appears to be to stop using suds-jurko and replace it with suds-bis or suds-community.

---

I had lexicon running for local development a few years ago, and at the time I had it installed using pip install --editable .. Would it be possible to install the latest version of lexicon with setuptools? I've tried, but I'm getting this confusing error:

  Running setup.py develop for dns-lexicon
    ERROR: Command errored out with exit status 1:
     command: /Users/quinn/src/lexicon/env/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/Users/quinn/src/lexicon/setup.py'"'"'; __file__='"'"'/Users/quinn/src/lexicon/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps
         cwd: /Users/quinn/src/lexicon/
    Complete output (3 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    ModuleNotFoundError: No module named 'setuptools'
    ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/quinn/src/lexicon/env/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/Users/quinn/src/lexicon/setup.py'"'"'; __file__='"'"'/Users/quinn/src/lexicon/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.

That is confusing because if I run pip install setuptools the response is:

Requirement already satisfied: setuptools in ./env/lib/python3.9/site-packages (58.0.4)

I've googled these errors and can't find any working solutions. I hope someone has some suggestions? Thanks!

[quinncomendant]

I posted a couple updates to my original comment, where I explain the problem and solution. TL;DR: it seems suds-jurko needs to be replaced by an alternative that is actively maintained and updated to work with setuptools >=58, such as suds-community.

[adferrand]

Hello ! For sure I am willing to remove a deprecated dependency from the dependency tree of lexicon.

As of now I am unable to reproduce the error tough. So we need to dig the issue a little deeper.

What I can say to you has of now is that pip install --editable . will not work. Basically poetry is an entire alternative to pip, so you are not supposed to use it, and poetry install -E full should work out of the box.

[quinncomendant]

Hi @adferrand!

Originally, I got the error on my MacBook, but I was able to reproduce it on a VM. You can use the following steps to consistently reproduce the issue:

1. Boot a VM with Debian 10. On GCP, I used something like this:

gcloud compute instances create lexicon --project=YOURPROJECT --zone=us-central1-c --machine-type=e2-micro --network-interface=network-tier=PREMIUM,subnet=default --maintenance-policy=MIGRATE --create-disk=auto-delete=yes,boot=yes,device-name=lexicon,image=projects/debian-cloud/global/images/debian-10-buster-v20210916,mode=rw,size=10,type=projects/YOURPROJECT/zones/us-central1-c/diskTypes/pd-balanced --no-shielded-secure-boot --shielded-vtpm --shielded-integrity-monitoring --reservation-affinity=any

2. Install git, python, and poetry:

sudo apt-get update
sudo apt-get install git python3 python3-distutils
curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python3
source $HOME/.poetry/env

3. Clone the repo and try to install:

git clone https://github.com/AnalogJ/lexicon
cd lexicon
poetry install -E full

You should get this error:

I found that pip install -U git+https://github.com/quinncomendant/lexicon.git does work fine to install my fork on my servers. I asked on the Poetry Discord server and they said, “pip can install poetry projects from source as long as you included the [build-system] section in the pyproject.toml as described here (This is done by default). Please note that pip will not take into account the poetry.lock, so it will resolve the dependencies by its own.” I'm just still confused why pip install works, but pip install --editable doesn't. 🤷‍♂️

[adferrand]

I see that now consistently on the CI. The faulty dependency is pulled from transip. Transip provider itself is using a deprecated SOAP API that will be removed soon.

To solve all of this I basically deprecated the transip provider in lexicon, disable the tests, in order to rebuild it on the new REST API.

This is live on 3.8.0.

[adferrand]

About the pip install editable, this is a known issue with pip when a project uses pyproject.toml instead ofnsetup.py.

I hope it will be fixed in the future, but as of now you should just use poetry which is basically with it that the dev environment is designed for lexicon.

[quinncomendant]

I can install 3.8.0 successfully, thanks @adferrand!