poetry config private repositories saves config to location conda-lock does not recognize

[leeleavitt]

Checklist

• I added a descriptive title
• I searched open reports and couldn't find a duplicate

What happened?

Description:

When following the directions for setting a private repository with Poetry using the command (this requires the user to install poetry, which cumbersome, and not required to lock non conda dependencies):

poetry config repositories.private_pypi $(PIP_INDEX_URL)

The config.toml file is created at ~/Library/Preferences/pypoetry.

However, conda-lock expects this config to be located at ~/Library/Application Support/pypoetry (see link1, link2).

As a workaround, the user can set the POETRY_CONFIG_DIR environment variable to ~/Library/Application Support/pypoetry as described in the Poetry documentation.

However, it would be more convenient and less error-prone if conda-lock could recognize the [[tool.poetry.source]] section in the pyproject.toml file, where private repository configurations are usually set:

[[tool.poetry.source]]
name = "my_software"
url = "https://pypi.hello.com/hi/prod/+simple"
default = true

Possible Solutions:

Update conda-lock to recognize the [[tool.poetry.source]] section in pyproject.toml.
Provide a conda-lock poetry-config option to set the private repository configuration.
Recommend a specific version of Poetry that sets the configuration in the Application Support directory, which is the location expected by conda-lock.

Conda Info

active environment : /Users/lleavitt/git/oligo_designer/.frak-venv
    active env location : /Users/lleavitt/git/oligo_designer/.frak-venv
            shell level : 2
       user config file : /Users/lleavitt/.condarc
 populated config files : /Users/lleavitt/opt/mambaforge/.condarc
                          /Users/lleavitt/.condarc
          conda version : 22.11.1
    conda-build version : not installed
         python version : 3.10.9.final.0
       virtual packages : __archspec=1=x86_64
                          __osx=12.6.3=0
                          __unix=0=0
       base environment : /Users/lleavitt/opt/mambaforge  (writable)
      conda av data dir : /Users/lleavitt/opt/mambaforge/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/osx-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /Users/lleavitt/opt/mambaforge/pkgs
                          /Users/lleavitt/.conda/pkgs
       envs directories : /Users/lleavitt/opt/mambaforge/envs
                          /Users/lleavitt/.conda/envs
               platform : osx-64
             user-agent : conda/22.11.1 requests/2.28.2 CPython/3.10.9 Darwin/21.6.0 OSX/12.6.3
                UID:GID : 1643950885:317021483
             netrc file : None
           offline mode : False

Conda Config

$ conda config --show-sources
==> /Users/lleavitt/opt/mambaforge/.condarc <==
channels:
  - conda-forge

==> /Users/lleavitt/.condarc <==
ssl_verify: False
channels:
  - conda-forge

Conda list

conda list --show-channel-urls
# packages in environment at /Users/lleavitt/git/oligo_designer/.frak-venv:
#
# Name                    Version                   Build  Channel
appdirs                   1.4.4                    pypi_0    pypi
arrow                     1.2.3                    pypi_0    pypi
attrs                     22.2.0             pyh71513ae_0    conda-forge
binaryornot               0.4.4                    pypi_0    pypi
brotlipy                  0.7.0           py38hef030d1_1005    conda-forge
bzip2                     1.0.8                h0d85af4_4    conda-forge
ca-certificates           2022.12.7            h033912b_0    conda-forge
cachecontrol              0.12.11            pyhd8ed1ab_1    conda-forge
cachy                     0.3.0                    pypi_0    pypi
certifi                   2022.12.7                pypi_0    pypi
cffi                      1.15.1           py38hb368cf1_3    conda-forge
chardet                   4.0.0                    pypi_0    pypi
charset-normalizer        2.0.12                   pypi_0    pypi
cleo                      2.0.1              pyhd8ed1ab_0    conda-forge
click                     8.0.4                    pypi_0    pypi
click-default-group       1.2.2                    pypi_0    pypi
clikit                    0.6.2                    pypi_0    pypi
colorama                  0.4.5                    pypi_0    pypi
conda-lock                1.3.0                    pypi_0    pypi
cookiecutter              1.7.3                    pypi_0    pypi
crashtest                 0.4.1              pyhd8ed1ab_0    conda-forge
crayons                   0.4.0                    pypi_0    pypi
cryptography              36.0.2                   pypi_0    pypi
defopt                    6.4.0                    pypi_0    pypi
distlib                   0.3.6              pyhd8ed1ab_0    conda-forge
docutils                  0.18.1                   pypi_0    pypi
dulwich                   0.20.50          py38hef030d1_0    conda-forge
dxpy                      0.340.1                  pypi_0    pypi
ensureconda               1.4.2                    pypi_0    pypi
filelock                  3.10.0             pyhd8ed1ab_0    conda-forge
frak                      0.10.0                   pypi_0    pypi
gitdb                     4.0.9                    pypi_0    pypi
gitpython                 3.1.18                   pypi_0    pypi
gnureadline               8.0.0                    pypi_0    pypi
html5lib                  1.1                pyh9f0ad1d_0    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
importlib-metadata        4.13.0             pyha770c72_0    conda-forge
importlib_metadata        4.13.0               hd8ed1ab_0    conda-forge
importlib_resources       5.12.0             pyhd8ed1ab_0    conda-forge
iniconfig                 1.1.1                    pypi_0    pypi
jaraco.classes            3.2.3              pyhd8ed1ab_0    conda-forge
jinja2                    3.0.3                    pypi_0    pypi
jinja2-time               0.2.0                    pypi_0    pypi
jsonschema                4.17.3             pyhd8ed1ab_0    conda-forge
keyring                   23.13.1          py38h50d1736_0    conda-forge
libblas                   3.9.0           16_osx64_openblas    conda-forge
libcblas                  3.9.0           16_osx64_openblas    conda-forge
libcxx                    15.0.7               h71dddab_0    conda-forge
libffi                    3.4.2                h0d85af4_5    conda-forge
libgfortran               5.0.0           11_3_0_h97931a8_31    conda-forge
libgfortran5              12.2.0              he409387_31    conda-forge
liblapack                 3.9.0           16_osx64_openblas    conda-forge
libopenblas               0.3.21          openmp_h429af6e_3    conda-forge
libsqlite                 3.40.0               ha978bb4_0    conda-forge
libzlib                   1.2.13               hfd90126_4    conda-forge
llvm-openmp               15.0.7               h61d9ccf_0    conda-forge
lockfile                  0.12.2                   pypi_0    pypi
markupsafe                2.0.1                    pypi_0    pypi
more-itertools            9.1.0              pyhd8ed1ab_0    conda-forge
msgpack                   1.0.5rc1                 pypi_0    pypi
msgpack-python            1.0.5            py38h98b9b1b_0    conda-forge
mypy-extensions           1.0.0                    pypi_0    pypi
ncurses                   6.3                  h96cf925_1    conda-forge
numpy                     1.24.2           py38h5a2dcdf_0    conda-forge
openssl                   3.1.0                hfd90126_0    conda-forge
packaging                 21.3                     pypi_0    pypi
pastel                    0.2.1                    pypi_0    pypi
pexpect                   4.8.0              pyh1a96a4e_2    conda-forge
pip                       23.0.1             pyhd8ed1ab_0    conda-forge
pkginfo                   1.9.6              pyhd8ed1ab_0    conda-forge
pkgutil-resolve-name      1.3.10             pyhd8ed1ab_0    conda-forge
platformdirs              2.6.2              pyhd8ed1ab_0    conda-forge
pluggy                    1.0.0                    pypi_0    pypi
pockets                   0.9.1                    pypi_0    pypi
poetry                    1.3.0            py38h50d1736_1    conda-forge
poetry-core               1.4.0              pyhd8ed1ab_0    conda-forge
poetry-plugin-export      1.3.0              pyhd8ed1ab_0    conda-forge
poyo                      0.5.0                    pypi_0    pypi
psutil                    5.9.4                    pypi_0    pypi
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
py                        1.11.0                   pypi_0    pypi
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pydantic                  1.9.2                    pypi_0    pypi
pylev                     1.4.0                    pypi_0    pypi
pyopenssl                 23.0.0             pyhd8ed1ab_0    conda-forge
pyparsing                 3.0.9                    pypi_0    pypi
pyproject_hooks           1.0.0              pyhd8ed1ab_0    conda-forge
pyrsistent                0.19.3           py38hef030d1_0    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
pytest                    7.0.1                    pypi_0    pypi
pytest-mock               3.6.1                    pypi_0    pypi
python                    3.8.16          hf9b03c3_1_cpython    conda-forge
python-build              0.10.0             pyhd8ed1ab_0    conda-forge
python-dateutil           2.8.2                    pypi_0    pypi
python-installer          0.6.0              pyhd8ed1ab_0    conda-forge
python-slugify            6.1.2                    pypi_0    pypi
python_abi                3.8                      3_cp38    conda-forge
pyyaml                    6.0                      pypi_0    pypi
rapidfuzz                 2.13.7           py38h4cd09af_0    conda-forge
readline                  8.1.2                h3899abd_0    conda-forge
requests                  2.26.0                   pypi_0    pypi
requests-toolbelt         0.10.1             pyhd8ed1ab_0    conda-forge
ruamel-yaml               0.17.21                  pypi_0    pypi
ruamel-yaml-clib          0.2.7                    pypi_0    pypi
setuptools                67.6.0             pyhd8ed1ab_0    conda-forge
shellingham               1.5.0.post1              pypi_0    pypi
six                       1.16.0             pyh6c4a22f_0    conda-forge
smmap                     5.0.0                    pypi_0    pypi
sphinxcontrib-napoleon    0.7                      pypi_0    pypi
text-unidecode            1.3                      pypi_0    pypi
tk                        8.6.12               h5dbffcc_0    conda-forge
toml                      0.10.2                   pypi_0    pypi
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
tomlkit                   0.11.6             pyha770c72_0    conda-forge
toolz                     0.12.0                   pypi_0    pypi
trove-classifiers         2023.3.9           pyhd8ed1ab_0    conda-forge
typing                    3.10.0.0           pyhd8ed1ab_0    conda-forge
typing-extensions         4.1.1                    pypi_0    pypi
typing-inspect            0.8.0                    pypi_0    pypi
typing_extensions         4.5.0              pyha770c72_0    conda-forge
urllib3                   1.26.14                  pypi_0    pypi
virtualenv                20.17.1                  pypi_0    pypi
webencodings              0.5.1                    pypi_0    pypi
websocket-client          0.54.0                   pypi_0    pypi
wheel                     0.40.0             pyhd8ed1ab_0    conda-forge
xattr                     0.10.1           py38hef030d1_0    conda-forge
xz                        5.2.6                h775f41a_0    conda-forge
zipp                      3.6.0                    pypi_0    pypi

Additional Context

No response

[maresb]

Hi @leeleavitt, thanks a lot for the feedback!

this requires the user to install poetry, which cumbersome, and not required to lock non conda dependencies

Agreed.

However, conda-lock expects this config to be located at...

There have been some substantial updates to Poetry since we vendored it. It's due for an update (#310). In particular, recent versions of Poetry seem to be using user_config_path from the platformdirs package (as opposed to the functions from your links), which probably leads to the discrepancy you note.

However, it would be more convenient and less error-prone if conda-lock could recognize the [[tool.poetry.source]] section in the pyproject.toml file

Yes, there is some great work in this direction by @croth1-liveeo, sitting in the review queue at #353. It would be of great help if you could check out that branch and see if it already accomplishes what you have in mind, and to provide feedback there.