Support Poetry

[danielknell]

It would be nice to be able to build docs for libraries managed with poetry.

maybe by checking for a [tool.poetry] section in pyproject.toml, having to keep a separate requirements.txt file in sync feels a bit fragile.

[stsewd]

What would be the poetry workflow in read the docs? Pipfile support will be out soon btw. If this gets implemented, it would live as an option in the configuration file.

[danielknell]

locally i run poetry install to create a virtualenv and install the dependencies, then poetry run sphinx-build ... to run sphinx-build inside that virtualenv.

although if there is a virtualenv already active then poetry will install the dependencies into that instead, which means it should work the same as for requirements.txt or Pipfile?

[sobolevn]

My workflow is pretty similar:

1. poetry install, which installs both dev and prod dependencies by default
2. cd docs, optional step
3. poetry run make html, to build the static website

Real project example that uses both poetry and rtd: https://github.com/wemake-services/wemake-django-template

[humitos]

So, replacing the pip install step by poetry install is all that's needed here or there are more tricks that I'm not seeing?

[sobolevn]

@humitos yes, I think so.

Mind, that we also have to work with venv differently: we need to prefix all commands with poetry run.

[danielknell]

if you have the venv active, you can run poetry install and it will install into the current environment the same way that pip install -r requirements.txt will, you can then use the venv as you would had normally.

poetry run is just a shorthand to run things in the venv without having to remember where it is or activate it first.

[stsewd]

dev and prod dependencies

Is there a use case where users only want to install dev/prod deps? There are any other configurations that users may want to set?

[danielknell]

poetry install will install both dev and regular dependencies
poetry install --no-dev will install only regular dependencies and no dev stuff

the latter is more along the lines of setup.py install with the caveat that it wont install the module itself, i cant think of a reason off the top of my head not to install dev dependencies for docs generation...

you can also use poetry install --extras "foo bar" to install extra dependency groups foo and bar

full docs are at https://poetry.eustace.io/docs/cli/#install

[uSpike]

I think this problem will resolve itself once the next version of pip is released with PEP-517 support (I've heard in January 2019?). It was merged - pypa/pip#5743
Then, a project only needs to add

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

to the pyproject.toml, and the normal pip install should work.

[danielknell]

except the build script runs python setup.py install --force by default instead of a pip install . so pep-517 wont come into play?

[stsewd]

@danielknell we do support installing the project using pip.

@uSpike I have read the pip issue and pep, but I wasn't able to understand it very well yet :/ will this replace pipenv too?

[uSpike]

@stsewd pip might be able to install from a Pipfile at some point in the future, but AFAIK it is not on the road map any time soon.

[cjw296]

@danielknell - yeah, check here: https://docs.readthedocs.io/en/latest/yaml-config.html#python-pip-install, might be able to do it through the web config too. So, guess we're just waiting on pypa/pip#6106. @stsewd : once pip 0.19.0 is out, how long before it makes it into a readthedocs image that we can use?

[stsewd]

We install the latest version of pip before each build #4938, so it should be available after they release it :)

[GPMueller]

I'm not very familiar with Python virtual environments, but there may be another issue there.

I run poetry from a conf.py script, but get the following error, which I don't see how it could be fixed from user side:

Creating virtualenv clang-api-doc-py3.5 in /home/docs/.cache/pypoetry/virtualenvs
The virtual environment was not created successfully because ensurepip is not
available.  On Debian/Ubuntu systems, you need to install the python3-venv
package using the following command.

    apt-get install python3-venv

You may need to use sudo with that command.  After installing the python3-venv
package, recreate your virtual environment.

Failing command: ['/home/docs/.cache/pypoetry/virtualenvs/clang-api-doc-py3.5/bin/python', '-Im', 'ensurepip', '--upgrade', '--default-pip']

[lorinkoz]

pypa/pip#6106 (comment)

[sobolevn]

Since new pip@19 is release, here's the short overview of what is going on.
I have locally checked how pip@19 works with two setups:

1. Application: https://github.com/wemake-services/wemake-django-template
2. Library: https://github.com/wemake-services/wemake-python-styleguide

Application

Trying: pip install .
Result:

    poetry.masonry.utils.module.ModuleOrPackageNotFound: No file/folder found for package wemake-django-template

    ----------------------------------------
Command "/Users/sobolev/Documents/github/wemake-django-template/.venv/bin/python3 /Users/sobolev/Documents/github/wemake-django-template/.venv/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/qn/2gssw9hx48g81chw0398hlrr0000gn/T/tmpbzmmkx48" failed with error code 1 in /private/var/folders/qn/2gssw9hx48g81chw0398hlrr0000gn/T/pip-req-build-2qbtac_8

Library

Trying: pip install .
Result: It installs everything as it should.

[danielknell]

does it work within rtfd? i keep meaning to try but have not found time to update my projects yet with the config to install via pip.

[uSpike]

@sobolevn I wouldn't expect https://github.com/wemake-services/wemake-django-template to install since it is a template (cookiecutter) project.

[sobolevn]

@uSpike poetry install works, so I had expected pip install to work as well.

[danielknell]

poetry install will only install dev/prod dependencies, not the module itself
pip install . will try and install the module itself, and with it the prod dependencies.

the fact it does not install dev dependencies is concerning and maybe pep 517 isn't enough to fully support poetry projects?

[sobolevn]

poetry install also installs the project itself in the editable mode. https://github.com/sdispater/poetry/releases/tag/0.12.0

I have also tried to research this dev/no-dev problem, looks like there is no known solution yet.

[danielknell]

that means poetry install is equivalent to pip install -e .

[stsewd]

Can anyone try a project in rtd? We have the latest pip installed, let us know if there is anything else to do from our side to support poetry.

[uSpike]

@sobolevn that project (https://github.com/wemake-services/wemake-django-template) is a template, you're not supposed to pip install it. The equivalent is having a requirements.txt, but they used poetry instead. The actual project is inside https://github.com/wemake-services/wemake-django-template/tree/master/%7B%7Bcookiecutter.project_name%7D%7D but you must run cookiecutter to make it.

@danielknell aah I forgot about that... there might be a way to pass flags to poetry via pip?

[agjohnson]

unfortunately theres nothing really in pyproject.toml to say which version of poetry is needed

Ah I thought you were referencing something like build-system.requires. Either way, there might be something here.

i'll just note the following as its a bit confusing and non obvious unless you follow poetrys development closely.

And thanks again for another helpful overview. I might be the only core team member that uses Poetry consistently, but even my usage is mostly experimentation. So I'm certainly not up to speed on new features.

This is where I think most of the team has less appetite to support a new packaging tooling. We did go through this with Pip -- there are seemingly endless ways users use Pip. Coupled with the fact that we're not skilled with Poetry, it might be hard for us to support all of the usage patterns natively.

It's definitely a complex problem at our scale. We can't expect any specific version of Poetry, but that limits us from supporting features like dependency groups -- our implementation can't assume the user is using a modern Poetry.

[danielknell]

when its released, poetry 1.2 should be able to support any older setups, it will treat the existing [tool.poetry.dev-dependencies] as if you had a [tool.poetry.group.dev.dependencies] so projects using the older version will have a standard and dev group but newer ones will have standard plus what ever the user defined.

my experience over the years has been pretty good backwards compatibility wise.

tbh, the easy route is probably only supporting the latest stable version only, and then doing a vanilla poetry install (plus -E foo if extra_requirements: ["foo"] is configured) which will install all dependency groups by default... I don't think I've seen a usage that wouldn't work for eventually.

it's not the perfect ideal setup but it would work for pretty much everything using a stable version, and early adopters of 1.2 would just have to wait (I mean that's par for the course when you go out on the bleeding edge right?), or use build.jobs?

[danielknell]

that would give you something like:

version: 2

python:
  install:
    - method: poetry
      extra_requirements:
        - foo
        - bar

translating into something like POETRY_VIRTUALENVS_CREATE=false poetry install -E foo -E bar instead of pip install .[foo,bar]

[danielknell]

would an extension of the following with the snippet below work?

readthedocs.org/readthedocs/doc_builder/python_environments.py

Lines 190 to 224 in 4ad6da0

	if install.method == PIP:
	# Prefix ./ so pip installs from a local path rather than pypi
	local_path = (
	os.path.join('.', install.path) if install.path != '.' else install.path
	)
	extra_req_param = ''
	if install.extra_requirements:
	extra_req_param = '[{}]'.format(
	','.join(install.extra_requirements)
	)
	self.build_env.run(
	self.venv_bin(filename='python'),
	'-m',
	'pip',
	'install',
	'--upgrade',
	'--upgrade-strategy',
	'eager',
	'--no-cache-dir',
	'{path}{extra_requirements}'.format(
	path=local_path,
	extra_requirements=extra_req_param,
	),
	cwd=self.checkout_path,
	bin_path=self.venv_bin(),
	)
	elif install.method == SETUPTOOLS:
	self.build_env.run(
	self.venv_bin(filename='python'),
	os.path.join(install.path, 'setup.py'),
	'install',
	'--force',
	cwd=self.checkout_path,
	bin_path=self.venv_bin(),
	)

        elif install.method == POETRY:
            local_path = (
                os.path.join('.', install.path) if install.path != '.' else install.path
            )
            extra_req_param = ''
            if install.extra_requirements:
                extra_req_param = ["-E {}".format(v) for install.extra_requirements if v]
            self.build_env.run(
                'poetry',
                'install',
                *extra_req_param,
                cwd=os.path.join(self.checkout_path, local_path),
                bin_path="~/.local/bin",
                environment={
                  "VIRTUAL_ENV": self.venv_path(),
                  "PIP_NO_CACHE_DIR": "on",
                }
            )

I am probably missing something as I don't really know what I'm doing in this codebase, but it should be close enough for someone to help tidy up?

it assumes a curl -sSL https://install.python-poetry.org | python3 - was previously run on the build system to install poetry into its default location, probably better to use POETRY_HOME but I can't remember how that works off the top of my head... passing VIRTUAL_ENV environment var should override poetrys internal venv management iirc and the --no-cache-dir from the pip branch can be forced with an environment var, then it just needs to run the command with cwd being the folder the pyproject.toml is in, so combining local and checkout path should work there?

[trim21]

you may need to use this:

version: 2

build:
  os: "ubuntu-22.04"
  tools:
    python: "3.10"
  jobs:
    post_create_environment:
      # Install poetry
      # https://python-poetry.org/docs/#installing-manually
      - python -m pip install poetry
    post_install:
      # Install dependencies with 'docs' dependency group
      # https://python-poetry.org/docs/managing-dependencies/#dependency-groups
      - VIRTUAL_ENV=$READTHEDOCS_VIRTUALENV_PATH poetry install --with docs

sphinx:
  configuration: docs/conf.py