We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Describe the bug Dependency issue when building environment as of 8/26/2020
To Reproduce Steps to reproduce the behavior:
Expected behavior No dependency issues
Logs
Here's the output of step 4:
(fix-postproc-errs) [aparker@el1 buildstockbatch-fix-postproc-errs]$ pip install -e . Obtaining file:///lustre/eaglefs/projects/comstock/repos/buildstockbatch-fix-postproc-errs Requirement already satisfied: pyyaml in /lustre/eaglefs/shared-projects/buildstock/envs/fix-postproc-errs/lib/python3.7/site-packages (from buildstock-batch==0.18) (5.3.1) Collecting requests Using cached requests-2.24.0-py2.py3-none-any.whl (61 kB) Requirement already satisfied: numpy>=1.11 in /lustre/eaglefs/shared-projects/buildstock/envs/fix-postproc-errs/lib/python3.7/site-packages (from buildstock-batch==0.18) (1.19.1) Requirement already satisfied: pandas!=1.0.4,>=1.0.0 in /lustre/eaglefs/shared-projects/buildstock/envs/fix-postproc-errs/lib/python3.7/site-packages (from buildstock-batch==0.18) (1.1.1) Collecting joblib Using cached joblib-0.16.0-py3-none-any.whl (300 kB) Requirement already satisfied: pyarrow>=0.14.1 in /lustre/eaglefs/shared-projects/buildstock/envs/fix-postproc-errs/lib/python3.7/site-packages (from buildstock-batch==0.18) (1.0.1) Requirement already satisfied: dask[complete]>=2.1.0 in /lustre/eaglefs/shared-projects/buildstock/envs/fix-postproc-errs/lib/python3.7/site-packages (from buildstock-batch==0.18) (2.24.0) Collecting docker Using cached docker-4.3.1-py2.py3-none-any.whl (145 kB) Collecting boto3>=1.10.44 Downloading boto3-1.14.48-py2.py3-none-any.whl (129 kB) |████████████████████████████████| 129 kB 911 kB/s Collecting s3fs>=0.4.0 Using cached s3fs-0.5.0-py3-none-any.whl (21 kB) Requirement already satisfied: fsspec>=0.6.0 in /lustre/eaglefs/shared-projects/buildstock/envs/fix-postproc-errs/lib/python3.7/site-packages (from buildstock-batch==0.18) (0.8.0) Collecting docutils<0.15,>=0.10 Using cached docutils-0.14-py3-none-any.whl (543 kB) Processing /home/aparker/.cache/pip/wheels/26/51/78/d8eafd10407e5822f792def21f9a477168a24933fe10ea9b48/yamale-3.0.3-py3-none-any.whl Collecting ruamel.yaml>=0.15.0 Using cached ruamel.yaml-0.16.10-py2.py3-none-any.whl (111 kB) Collecting testfixtures Using cached testfixtures-6.14.1-py2.py3-none-any.whl (89 kB) Collecting awsretry Using cached awsretry-1.0.1-py2.py3-none-any.whl (6.5 kB) Requirement already satisfied: certifi>=2017.4.17 in /lustre/eaglefs/shared-projects/buildstock/envs/fix-postproc-errs/lib/python3.7/site-packages (from requests->buildstock-batch==0.18) (2020.6.20) Collecting chardet<4,>=3.0.2 Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB) Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 Using cached urllib3-1.25.10-py2.py3-none-any.whl (127 kB) Collecting idna<3,>=2.5 Using cached idna-2.10-py2.py3-none-any.whl (58 kB) Requirement already satisfied: python-dateutil>=2.7.3 in /lustre/eaglefs/shared-projects/buildstock/envs/fix-postproc-errs/lib/python3.7/site-packages (from pandas!=1.0.4,>=1.0.0->buildstock-batch==0.18) (2.8.1) Requirement already satisfied: pytz>=2017.2 in /lustre/eaglefs/shared-projects/buildstock/envs/fix-postproc-errs/lib/python3.7/site-packages (from pandas!=1.0.4,>=1.0.0->buildstock-batch==0.18) (2020.1) Requirement already satisfied: distributed>=2.0; extra == "complete" in /lustre/eaglefs/shared-projects/buildstock/envs/fix-postproc-errs/lib/python3.7/site-packages (from dask[complete]>=2.1.0->buildstock-batch==0.18) (2.24.0) Requirement already satisfied: bokeh>=1.0.0; extra == "complete" in /lustre/eaglefs/shared-projects/buildstock/envs/fix-postproc-errs/lib/python3.7/site-packages (from dask[complete]>=2.1.0->buildstock-batch==0.18) (2.2.0) Requirement already satisfied: cloudpickle>=0.2.2; extra == "complete" in /lustre/eaglefs/shared-projects/buildstock/envs/fix-postproc-errs/lib/python3.7/site-packages (from dask[complete]>=2.1.0->buildstock-batch==0.18) (1.6.0) Requirement already satisfied: partd>=0.3.10; extra == "complete" in /lustre/eaglefs/shared-projects/buildstock/envs/fix-postproc-errs/lib/python3.7/site-packages (from dask[complete]>=2.1.0->buildstock-batch==0.18) (1.1.0) Requirement already satisfied: toolz>=0.8.2; extra == "complete" in /lustre/eaglefs/shared-projects/buildstock/envs/fix-postproc-errs/lib/python3.7/site-packages (from dask[complete]>=2.1.0->buildstock-batch==0.18) (0.10.0) Collecting websocket-client>=0.32.0 Using cached websocket_client-0.57.0-py2.py3-none-any.whl (200 kB) Requirement already satisfied: six>=1.4.0 in /lustre/eaglefs/shared-projects/buildstock/envs/fix-postproc-errs/lib/python3.7/site-packages (from docker->buildstock-batch==0.18) (1.15.0) Collecting jmespath<1.0.0,>=0.7.1 Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB) Collecting s3transfer<0.4.0,>=0.3.0 Using cached s3transfer-0.3.3-py2.py3-none-any.whl (69 kB) Collecting botocore<1.18.0,>=1.17.48 Using cached botocore-1.17.48-py2.py3-none-any.whl (6.5 MB) Collecting aiobotocore>=1.0.1 Using cached aiobotocore-1.1.0-py3-none-any.whl (43 kB) Collecting ruamel.yaml.clib>=0.1.2; platform_python_implementation == "CPython" and python_version < "3.9" Using cached ruamel.yaml.clib-0.2.0-cp37-cp37m-manylinux1_x86_64.whl (547 kB) Collecting boto Using cached boto-2.49.0-py2.py3-none-any.whl (1.4 MB) Requirement already satisfied: click>=6.6 in /lustre/eaglefs/shared-projects/buildstock/envs/fix-postproc-errs/lib/python3.7/site-packages (from distributed>=2.0; extra == "complete"->dask[complete]>=2.1.0->buildstock-batch==0.18) (7.1.2) Requirement already satisfied: zict>=0.1.3 in /lustre/eaglefs/shared-projects/buildstock/envs/fix-postproc-errs/lib/python3.7/site-packages (from distributed>=2.0; extra == "complete"->dask[complete]>=2.1.0->buildstock-batch==0.18) (2.0.0) Requirement already satisfied: tblib>=1.6.0 in /lustre/eaglefs/shared-projects/buildstock/envs/fix-postproc-errs/lib/python3.7/site-packages (from distributed>=2.0; extra == "complete"->dask[complete]>=2.1.0->buildstock-batch==0.18) (1.6.0) Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /lustre/eaglefs/shared-projects/buildstock/envs/fix-postproc-errs/lib/python3.7/site-packages (from distributed>=2.0; extra == "complete"->dask[complete]>=2.1.0->buildstock-batch==0.18) (2.2.2) Requirement already satisfied: msgpack>=0.6.0 in /lustre/eaglefs/shared-projects/buildstock/envs/fix-postproc-errs/lib/python3.7/site-packages (from distributed>=2.0; extra == "complete"->dask[complete]>=2.1.0->buildstock-batch==0.18) (1.0.0) Requirement already satisfied: psutil>=5.0 in /lustre/eaglefs/shared-projects/buildstock/envs/fix-postproc-errs/lib/python3.7/site-packages (from distributed>=2.0; extra == "complete"->dask[complete]>=2.1.0->buildstock-batch==0.18) (5.7.2) Requirement already satisfied: tornado>=5; python_version < "3.8" in /lustre/eaglefs/shared-projects/buildstock/envs/fix-postproc-errs/lib/python3.7/site-packages (from distributed>=2.0; extra == "complete"->dask[complete]>=2.1.0->buildstock-batch==0.18) (6.0.4) Requirement already satisfied: setuptools in /lustre/eaglefs/shared-projects/buildstock/envs/fix-postproc-errs/lib/python3.7/site-packages (from distributed>=2.0; extra == "complete"->dask[complete]>=2.1.0->buildstock-batch==0.18) (49.6.0.post20200814) Requirement already satisfied: Jinja2>=2.7 in /lustre/eaglefs/shared-projects/buildstock/envs/fix-postproc-errs/lib/python3.7/site-packages (from bokeh>=1.0.0; extra == "complete"->dask[complete]>=2.1.0->buildstock-batch==0.18) (2.11.2) Requirement already satisfied: typing-extensions>=3.7.4 in /lustre/eaglefs/shared-projects/buildstock/envs/fix-postproc-errs/lib/python3.7/site-packages (from bokeh>=1.0.0; extra == "complete"->dask[complete]>=2.1.0->buildstock-batch==0.18) (3.7.4.2) Requirement already satisfied: pillow>=7.1.0 in /lustre/eaglefs/shared-projects/buildstock/envs/fix-postproc-errs/lib/python3.7/site-packages (from bokeh>=1.0.0; extra == "complete"->dask[complete]>=2.1.0->buildstock-batch==0.18) (7.2.0) Requirement already satisfied: packaging>=16.8 in /lustre/eaglefs/shared-projects/buildstock/envs/fix-postproc-errs/lib/python3.7/site-packages (from bokeh>=1.0.0; extra == "complete"->dask[complete]>=2.1.0->buildstock-batch==0.18) (20.4) Requirement already satisfied: locket in /lustre/eaglefs/shared-projects/buildstock/envs/fix-postproc-errs/lib/python3.7/site-packages (from partd>=0.3.10; extra == "complete"->dask[complete]>=2.1.0->buildstock-batch==0.18) (0.2.0) Processing /home/aparker/.cache/pip/wheels/62/76/4c/aa25851149f3f6d9785f6c869387ad82b3fd37582fa8147ac6/wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl Collecting aiohttp>=3.3.1 Using cached aiohttp-3.6.2-cp37-cp37m-manylinux1_x86_64.whl (1.2 MB) Collecting aioitertools>=0.5.1 Using cached aioitertools-0.7.0-py3-none-any.whl (20 kB) Requirement already satisfied: heapdict in /lustre/eaglefs/shared-projects/buildstock/envs/fix-postproc-errs/lib/python3.7/site-packages (from zict>=0.1.3->distributed>=2.0; extra == "complete"->dask[complete]>=2.1.0->buildstock-batch==0.18) (1.0.1) Requirement already satisfied: MarkupSafe>=0.23 in /lustre/eaglefs/shared-projects/buildstock/envs/fix-postproc-errs/lib/python3.7/site-packages (from Jinja2>=2.7->bokeh>=1.0.0; extra == "complete"->dask[complete]>=2.1.0->buildstock-batch==0.18) (1.1.1) Requirement already satisfied: pyparsing>=2.0.2 in /lustre/eaglefs/shared-projects/buildstock/envs/fix-postproc-errs/lib/python3.7/site-packages (from packaging>=16.8->bokeh>=1.0.0; extra == "complete"->dask[complete]>=2.1.0->buildstock-batch==0.18) (2.4.7) Collecting yarl<2.0,>=1.0 Using cached yarl-1.5.1-cp37-cp37m-manylinux1_x86_64.whl (258 kB) Collecting multidict<5.0,>=4.5 Using cached multidict-4.7.6-cp37-cp37m-manylinux1_x86_64.whl (149 kB) Collecting attrs>=17.3.0 Using cached attrs-20.1.0-py2.py3-none-any.whl (49 kB) Collecting async-timeout<4.0,>=3.0 Using cached async_timeout-3.0.1-py3-none-any.whl (8.2 kB) Installing collected packages: chardet, urllib3, idna, requests, joblib, websocket-client, docker, jmespath, docutils, botocore, s3transfer, boto3, wrapt, multidict, yarl, attrs, async-timeout, aiohttp, aioitertools, aiobotocore, s3fs, Running setup.py develop for buildstock-batchtures, boto, awsretry, buildstock-batch ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts. We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default. aiobotocore 1.1.0 requires botocore<1.17.45,>=1.17.44, but you'll have botocore 1.17.48 which is incompatible. Successfully installed aiobotocore-1.1.0 aiohttp-3.6.2 aioitertools-0.7.0 async-timeout-3.0.1 attrs-20.1.0 awsretry-1.0.1 boto-2.49.0 boto3-1.14.48 botocore-1.17.48 buildstock-batch chardet-3.0.4 docker-4.3.1 docutils-0.14 idna-2.10 jmespath-0.10.0 joblib-0.16.0 multidict-4.7.6 requests-2.24.0 ruamel.yaml-0.16.10 ruamel.yaml.clib-0.2.0 s3fs-0.5.0 s3transfer-0.3.3 testfixtures-6.14.1 urllib3-1.25.10 websocket-client-0.57.0 wrapt-1.12.1 yamale-3.0.3 yarl-1.5.1
And here's the output of pipdeptree
pipdeptree
(fix-postproc-errs) [aparker@el1 buildstockbatch-fix-postproc-errs]$ pipdeptree Warning!!! Possibly conflicting dependencies found: * aiobotocore==1.1.0 - botocore [required: >=1.17.44,<1.17.45, installed: 1.17.48] ------------------------------------------------------------------------ bokeh==2.2.0 - Jinja2 [required: >=2.7, installed: 2.11.2] - MarkupSafe [required: >=0.23, installed: 1.1.1] - numpy [required: >=1.11.3, installed: 1.19.1] - packaging [required: >=16.8, installed: 20.4] - pyparsing [required: >=2.0.2, installed: 2.4.7] - six [required: Any, installed: 1.15.0] - pillow [required: >=7.1.0, installed: 7.2.0] - python-dateutil [required: >=2.1, installed: 2.8.1] - six [required: >=1.5, installed: 1.15.0] - PyYAML [required: >=3.10, installed: 5.3.1] - tornado [required: >=5.1, installed: 6.0.4] - typing-extensions [required: >=3.7.4, installed: 3.7.4.2] buildstock-batch==0.18 - awsretry [required: Any, installed: 1.0.1] - boto [required: Any, installed: 2.49.0] - boto3 [required: Any, installed: 1.14.48] - botocore [required: >=1.17.48,<1.18.0, installed: 1.17.48] - docutils [required: >=0.10,<0.16, installed: 0.14] - jmespath [required: >=0.7.1,<1.0.0, installed: 0.10.0] - python-dateutil [required: >=2.1,<3.0.0, installed: 2.8.1] - six [required: >=1.5, installed: 1.15.0] - urllib3 [required: >=1.20,<1.26, installed: 1.25.10] - jmespath [required: >=0.7.1,<1.0.0, installed: 0.10.0] - s3transfer [required: >=0.3.0,<0.4.0, installed: 0.3.3] - botocore [required: >=1.12.36,<2.0a.0, installed: 1.17.48] - docutils [required: >=0.10,<0.16, installed: 0.14] - jmespath [required: >=0.7.1,<1.0.0, installed: 0.10.0] - python-dateutil [required: >=2.1,<3.0.0, installed: 2.8.1] - six [required: >=1.5, installed: 1.15.0] - urllib3 [required: >=1.20,<1.26, installed: 1.25.10] - botocore [required: Any, installed: 1.17.48] - docutils [required: >=0.10,<0.16, installed: 0.14] - jmespath [required: >=0.7.1,<1.0.0, installed: 0.10.0] - python-dateutil [required: >=2.1,<3.0.0, installed: 2.8.1] - six [required: >=1.5, installed: 1.15.0] - urllib3 [required: >=1.20,<1.26, installed: 1.25.10] - boto3 [required: >=1.10.44, installed: 1.14.48] - botocore [required: >=1.17.48,<1.18.0, installed: 1.17.48] - docutils [required: >=0.10,<0.16, installed: 0.14] - jmespath [required: >=0.7.1,<1.0.0, installed: 0.10.0] - python-dateutil [required: >=2.1,<3.0.0, installed: 2.8.1] - six [required: >=1.5, installed: 1.15.0] - urllib3 [required: >=1.20,<1.26, installed: 1.25.10] - jmespath [required: >=0.7.1,<1.0.0, installed: 0.10.0] - s3transfer [required: >=0.3.0,<0.4.0, installed: 0.3.3] - botocore [required: >=1.12.36,<2.0a.0, installed: 1.17.48] - docutils [required: >=0.10,<0.16, installed: 0.14] - jmespath [required: >=0.7.1,<1.0.0, installed: 0.10.0] - python-dateutil [required: >=2.1,<3.0.0, installed: 2.8.1] - six [required: >=1.5, installed: 1.15.0] - urllib3 [required: >=1.20,<1.26, installed: 1.25.10] - dask [required: >=2.1.0, installed: 2.24.0] - pyyaml [required: Any, installed: 5.3.1] - docker [required: Any, installed: 4.3.1] - requests [required: >=2.14.2,!=2.18.0, installed: 2.24.0] - certifi [required: >=2017.4.17, installed: 2020.6.20] - chardet [required: >=3.0.2,<4, installed: 3.0.4] - idna [required: >=2.5,<3, installed: 2.10] - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.10] - six [required: >=1.4.0, installed: 1.15.0] - websocket-client [required: >=0.32.0, installed: 0.57.0] - six [required: Any, installed: 1.15.0] - docutils [required: >=0.10,<0.15, installed: 0.14] - fsspec [required: >=0.6.0, installed: 0.8.0] - joblib [required: Any, installed: 0.16.0] - numpy [required: >=1.11, installed: 1.19.1] - pandas [required: >=1.0.0,!=1.0.4, installed: 1.1.1] - numpy [required: >=1.15.4, installed: 1.19.1] - python-dateutil [required: >=2.7.3, installed: 2.8.1] - six [required: >=1.5, installed: 1.15.0] - pytz [required: >=2017.2, installed: 2020.1] - pyarrow [required: >=0.14.1, installed: 1.0.1] - numpy [required: >=1.14, installed: 1.19.1] - pyyaml [required: Any, installed: 5.3.1] - requests [required: Any, installed: 2.24.0] - certifi [required: >=2017.4.17, installed: 2020.6.20] - chardet [required: >=3.0.2,<4, installed: 3.0.4] - idna [required: >=2.5,<3, installed: 2.10] - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.10] - ruamel.yaml [required: >=0.15.0, installed: 0.16.10] - ruamel.yaml.clib [required: >=0.1.2, installed: 0.2.0] - s3fs [required: >=0.4.0, installed: 0.5.0] - aiobotocore [required: >=1.0.1, installed: 1.1.0] - aiohttp [required: >=3.3.1, installed: 3.6.2] - async-timeout [required: >=3.0,<4.0, installed: 3.0.1] - attrs [required: >=17.3.0, installed: 20.1.0] - chardet [required: >=2.0,<4.0, installed: 3.0.4] - multidict [required: >=4.5,<5.0, installed: 4.7.6] - yarl [required: >=1.0,<2.0, installed: 1.5.1] - idna [required: >=2.0, installed: 2.10] - multidict [required: >=4.0, installed: 4.7.6] - typing-extensions [required: >=3.7.4, installed: 3.7.4.2] - aioitertools [required: >=0.5.1, installed: 0.7.0] - typing-extensions [required: >=3.7, installed: 3.7.4.2] - botocore [required: >=1.17.44,<1.17.45, installed: 1.17.48] - docutils [required: >=0.10,<0.16, installed: 0.14] - jmespath [required: >=0.7.1,<1.0.0, installed: 0.10.0] - python-dateutil [required: >=2.1,<3.0.0, installed: 2.8.1] - six [required: >=1.5, installed: 1.15.0] - urllib3 [required: >=1.20,<1.26, installed: 1.25.10] - wrapt [required: >=1.10.10, installed: 1.12.1] - fsspec [required: >=0.8.0, installed: 0.8.0] - testfixtures [required: Any, installed: 6.14.1] - yamale [required: >=2.0, installed: 3.0.3] - pyyaml [required: Any, installed: 5.3.1] cytoolz==0.10.1 - toolz [required: >=0.8.0, installed: 0.10.0] distributed==2.24.0 - click [required: >=6.6, installed: 7.1.2] - cloudpickle [required: >=1.5.0, installed: 1.6.0] - dask [required: >=2.9.0, installed: 2.24.0] - pyyaml [required: Any, installed: 5.3.1] - msgpack [required: >=0.6.0, installed: 1.0.0] - psutil [required: >=5.0, installed: 5.7.2] - pyyaml [required: Any, installed: 5.3.1] - setuptools [required: Any, installed: 49.6.0.post20200814] - sortedcontainers [required: !=2.0.1,!=2.0.0, installed: 2.2.2] - tblib [required: >=1.6.0, installed: 1.6.0] - toolz [required: >=0.8.2, installed: 0.10.0] - tornado [required: >=5, installed: 6.0.4] - zict [required: >=0.1.3, installed: 2.0.0] - heapdict [required: Any, installed: 1.0.1] olefile==0.46 partd==1.1.0 - locket [required: Any, installed: 0.2.0] - toolz [required: Any, installed: 0.10.0] pipdeptree==1.0.0 - pip [required: >=6.0.0, installed: 20.2.2] wheel==0.35.1
Possible Resolution Looks like s3fs released a version 0.5.0, but the buildstock-0.18 used 0.4.2. Here's the dependency tree in question:
s3fs
0.5.0
buildstock-0.18
0.4.2
- s3fs [required: >=0.4.0, installed: 0.5.0] - aiobotocore [required: >=1.0.1, installed: 1.1.0] - botocore [required: >=1.17.44,<1.17.45, installed: 1.17.48]
I think the resolution is changing the setup.py file to: 's3fs>=0.4.0,<0.5.0',
setup.py
's3fs>=0.4.0,<0.5.0',
Platform (please complete the following information):
The text was updated successfully, but these errors were encountered:
@nmerket FYI
Sorry, something went wrong.
Yeah, it's either that or downgrading to a version of boto3 that uses an older version of botocore.
Also, I wasn't aware of pipdeptree. Very handy!
Successfully merging a pull request may close this issue.
Describe the bug
Dependency issue when building environment as of 8/26/2020
To Reproduce
Steps to reproduce the behavior:
Expected behavior
No dependency issues
Logs
Here's the output of step 4:
And here's the output of
pipdeptree
Possible Resolution
Looks like
s3fs
released a version0.5.0
, but thebuildstock-0.18
used0.4.2
. Here's the dependency tree in question:I think the resolution is changing the
setup.py
file to:'s3fs>=0.4.0,<0.5.0',
Platform (please complete the following information):
The text was updated successfully, but these errors were encountered: