Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CondaPackError: Files managed by conda were found to have been deleted/overwritten in the following packages: #197

Open
PolarisRisingWar opened this issue Nov 26, 2021 · 7 comments
Labels
stale::recovered [bot] recovered after being marked as stale

Comments

@PolarisRisingWar
Copy link

I've using this command: conda pack -n envesca -o (mypath)/condapacks/envesca.tar.gz

And I met this error message:

Collecting packages...
CondaPackError: 
Files managed by conda were found to have been deleted/overwritten in the
following packages:

- numpy 1.19.5:
    lib/python3.6/site-packages/numpy-1.19.5.dist-info/REQUESTED
    lib/python3.6/site-packages/numpy-1.19.5.dist-info/direct_url.json
    lib/python3.6/site-packages/numpy/distutils/site.cfg

This is usually due to `pip` uninstalling or clobbering conda managed files,
resulting in an inconsistent environment. Please check your environment for
conda/pip conflicts using `conda list`, and fix the environment by ensuring
only one version of each package is installed (conda preferred)

Python 3.6.13
Ubuntu

This is my conda list:

_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       1_gnu    conda-forge
absl-py                   1.0.0              pyhd8ed1ab_0    conda-forge
alembic                   1.7.5                    pypi_0    pypi
async-generator           1.10                     pypi_0    pypi
attrs                     21.2.0             pyhd8ed1ab_0    conda-forge
backports                 1.0                        py_2    conda-forge
backports.functools_lru_cache 1.6.4              pyhd8ed1ab_0    conda-forge
bleach                    4.1.0              pyhd8ed1ab_0    conda-forge
blis                      0.7.5                    pypi_0    pypi
boto3                     1.20.11                  pypi_0    pypi
botocore                  1.23.11                  pypi_0    pypi
c-ares                    1.18.1               h7f98852_0    conda-forge
ca-certificates           2021.10.26           h06a4308_2  
cachetools                4.2.4                    pypi_0    pypi
catalogue                 2.0.6                    pypi_0    pypi
certifi                   2021.10.8                pypi_0    pypi
certipy                   0.1.3                    pypi_0    pypi
cffi                      1.14.6           py36h400218f_0  
charset-normalizer        2.0.7                    pypi_0    pypi
click                     8.0.3                    pypi_0    pypi
conda-pack                0.6.0              pyhd3eb1b0_0  
contextvars               2.4                      pypi_0    pypi
coverage                  4.0.3                    py36_1    conda-forge
cryptography              36.0.0                   pypi_0    pypi
cudatoolkit               10.0.130             hf841e97_9    conda-forge
cupy-cuda100              9.6.0                    pypi_0    pypi
cymem                     2.0.6                    pypi_0    pypi
dataclasses               0.8                pyh787bdff_2    conda-forge
decorator                 5.1.0              pyhd8ed1ab_0    conda-forge
defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
dill                      0.3.4                    pypi_0    pypi
en-core-web-sm            3.2.0                    pypi_0    pypi
entrypoints               0.3             pyhd8ed1ab_1003    conda-forge
fastnlp                   0.7.0                    pypi_0    pypi
fastrlock                 0.8                      pypi_0    pypi
filelock                  3.4.0                    pypi_0    pypi
freetype                  2.10.4               h0708190_1    conda-forge
google-auth               2.3.3                    pypi_0    pypi
google-auth-oauthlib      0.4.6                    pypi_0    pypi
greenlet                  1.1.2                    pypi_0    pypi
grpcio                    1.42.0                   pypi_0    pypi
huggingface-hub           0.1.2                    pypi_0    pypi
idna                      3.3                      pypi_0    pypi
immutables                0.16                     pypi_0    pypi
importlib-metadata        4.8.2                    pypi_0    pypi
importlib-resources       5.4.0                    pypi_0    pypi
intel-openmp              2021.4.0          h06a4308_3561  
ipykernel                 4.9.0                    py36_1  
ipython                   5.8.0                    py36_1    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
jbig                      2.1               h7f98852_2003    conda-forge
jieba                     0.42.1                   pypi_0    pypi
jinja2                    3.0.3              pyhd8ed1ab_0    conda-forge
jmespath                  0.10.0                   pypi_0    pypi
joblib                    1.1.0                    pypi_0    pypi
jpeg                      9d                   h36c2ea0_0    conda-forge
jsonschema                3.2.0              pyhd8ed1ab_3    conda-forge
jupyter-telemetry         0.1.0                    pypi_0    pypi
jupyter_client            5.3.4                    py36_0    conda-forge
jupyter_core              4.5.0                      py_0    conda-forge
jupyterhub                1.5.0                    pypi_0    pypi
langcodes                 3.3.0                    pypi_0    pypi
lcms2                     2.12                 hddcbb42_0    conda-forge
ld_impl_linux-64          2.36.1               hea4e1c9_2    conda-forge
lerc                      3.0                  h9c3ff4c_0    conda-forge
libblas                   3.9.0           12_linux64_openblas    conda-forge
libcblas                  3.9.0           12_linux64_openblas    conda-forge
libdeflate                1.8                  h7f98852_0    conda-forge
libffi                    3.3                  h58526e2_2    conda-forge
libgcc-ng                 11.2.0              h1d223b6_11    conda-forge
libgfortran-ng            11.2.0              h69a702a_11    conda-forge
libgfortran5              11.2.0              h5c6108e_11    conda-forge
libgomp                   11.2.0              h1d223b6_11    conda-forge
liblapack                 3.9.0           12_linux64_openblas    conda-forge
libopenblas               0.3.18          pthreads_h8fe5266_0    conda-forge
libpng                    1.6.37               h21135ba_2    conda-forge
libprotobuf               3.15.8               h780b84a_1    conda-forge
libsodium                 1.0.18               h36c2ea0_1    conda-forge
libstdcxx-ng              11.2.0              he4da1e4_11    conda-forge
libtiff                   4.3.0                h6f004c6_2    conda-forge
libwebp-base              1.2.1                h7f98852_0    conda-forge
libzlib                   1.2.11            h36c2ea0_1013    conda-forge
llvm-openmp               12.0.1               h4bd325d_1    conda-forge
lz4-c                     1.9.3                h9c3ff4c_1    conda-forge
mako                      1.1.6                    pypi_0    pypi
markdown                  3.3.6                    pypi_0    pypi
markupsafe                2.0.1            py36h8f6f2f9_0    conda-forge
mistune                   0.8.4           py36h8f6f2f9_1004    conda-forge
mkl                       2021.4.0           h06a4308_640  
multiprocess              0.70.12.2                pypi_0    pypi
murmurhash                1.0.6                    pypi_0    pypi
nbconvert                 5.4.1                    py36_3  
nbformat                  5.1.3              pyhd8ed1ab_0    conda-forge
ncurses                   6.2                  h58526e2_4    conda-forge
ninja                     1.10.2               h4bd325d_1    conda-forge
notebook                  5.7.11           py36h5fab9bb_0    conda-forge
numpy                     1.19.5           py36hfc0c790_2    conda-forge
oauthlib                  3.1.1                    pypi_0    pypi
objgraph                  3.5.0                    pypi_0    pypi
olefile                   0.46               pyh9f0ad1d_1    conda-forge
openssl                   1.1.1l               h7f8727e_0  
packaging                 21.3               pyhd8ed1ab_0    conda-forge
paddlepaddle-tiny         1.6.1                    pypi_0    pypi
pamela                    1.0.0                    pypi_0    pypi
pandas                    1.1.5                    pypi_0    pypi
pandoc                    2.16.2               h7f98852_0    conda-forge
pandocfilters             1.5.0              pyhd8ed1ab_0    conda-forge
pathy                     0.6.1                    pypi_0    pypi
pexpect                   4.8.0              pyh9f0ad1d_2    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pillow                    7.2.0            py36h8328e55_2    conda-forge
pip                       21.3.1             pyhd8ed1ab_0    conda-forge
preshed                   3.0.6                    pypi_0    pypi
prettytable               2.4.0                    pypi_0    pypi
prometheus_client         0.12.0             pyhd8ed1ab_0    conda-forge
prompt_toolkit            1.0.15                     py_1    conda-forge
protobuf                  3.19.1                   pypi_0    pypi
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pyasn1                    0.4.8                    pypi_0    pypi
pyasn1-modules            0.2.8                    pypi_0    pypi
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pydantic                  1.8.2                    pypi_0    pypi
pygments                  2.10.0             pyhd8ed1ab_0    conda-forge
pyopenssl                 21.0.0                   pypi_0    pypi
pyparsing                 3.0.6              pyhd8ed1ab_0    conda-forge
pyrouge                   0.1.3                    pypi_0    pypi
pyrsistent                0.18.0                   pypi_0    pypi
python                    3.6.13          hffdb5ce_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-graphviz           0.18.2                   pypi_0    pypi
python-json-logger        2.0.2                    pypi_0    pypi
python-levenshtein        0.12.2                   pypi_0    pypi
python_abi                3.6                     2_cp36m    conda-forge
pythonrouge               0.2                      pypi_0    pypi
pytorch                   1.1.0           py3.6_cuda10.0.130_cudnn7.5.1_0    pytorch
pytorch-transformers      1.2.0                    pypi_0    pypi
pytz                      2021.3                   pypi_0    pypi
pyyaml                    6.0                      pypi_0    pypi
pyzmq                     22.3.0           py36h7068817_0    conda-forge
readline                  8.1                  h46c0cb4_0    conda-forge
regex                     2021.11.10               pypi_0    pypi
requests                  2.26.0                   pypi_0    pypi
requests-oauthlib         1.3.0                    pypi_0    pypi
rouge-metric              1.0.1                    pypi_0    pypi
rsa                       4.7.2                    pypi_0    pypi
ruamel-yaml               0.17.17                  pypi_0    pypi
ruamel-yaml-clib          0.2.6                    pypi_0    pypi
s3transfer                0.5.0                    pypi_0    pypi
sacremoses                0.0.46                   pypi_0    pypi
send2trash                1.8.0              pyhd8ed1ab_0    conda-forge
sentencepiece             0.1.96                   pypi_0    pypi
setuptools                59.2.0                   pypi_0    pypi
simplegeneric             0.8.1                      py_1    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
smart-open                5.2.1                    pypi_0    pypi
spacy                     3.2.0                    pypi_0    pypi
spacy-legacy              3.0.8                    pypi_0    pypi
spacy-loggers             1.0.1                    pypi_0    pypi
sqlalchemy                1.4.27                   pypi_0    pypi
sqlite                    3.36.0               h9cd32fc_2    conda-forge
srsly                     2.4.2                    pypi_0    pypi
tensorboard               2.7.0                    pypi_0    pypi
tensorboard-data-server   0.6.1                    pypi_0    pypi
tensorboard-plugin-wit    1.8.0                    pypi_0    pypi
tensorboardx              2.4.1                    pypi_0    pypi
terminado                 0.12.1           py36h5fab9bb_0    conda-forge
testpath                  0.5.0              pyhd8ed1ab_0    conda-forge
thinc                     8.0.13                   pypi_0    pypi
tk                        8.6.11               h27826a3_1    conda-forge
tokenizers                0.10.3                   pypi_0    pypi
torch-tb-profiler         0.3.1                    pypi_0    pypi
torchvision               0.3.0           py36_cu10.0.130_1    pytorch
tornado                   6.1              py36h8f6f2f9_1    conda-forge
tqdm                      4.62.3                   pypi_0    pypi
traitlets                 4.3.3              pyhd8ed1ab_2    conda-forge
transformers              4.12.5                   pypi_0    pypi
typer                     0.4.0                    pypi_0    pypi
typing-extensions         3.10.0.2                 pypi_0    pypi
urllib3                   1.26.7                   pypi_0    pypi
wasabi                    0.8.2                    pypi_0    pypi
wcwidth                   0.2.5              pyh9f0ad1d_2    conda-forge
webencodings              0.5.1                      py_1    conda-forge
werkzeug                  2.0.2                    pypi_0    pypi
wheel                     0.37.0             pyhd8ed1ab_1    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
zeromq                    4.3.4                h9c3ff4c_1    conda-forge
zipp                      3.6.0              pyhd8ed1ab_0    conda-forge
zlib                      1.2.11            h36c2ea0_1013    conda-forge
zstd                      1.5.0                ha95c52a_0    conda-forge

And this is my pip list:

Package                       Version
----------------------------- ----------
absl-py                       1.0.0
alembic                       1.7.5
async-generator               1.10
attrs                         21.2.0
backports.functools-lru-cache 1.6.4
bleach                        4.1.0
blis                          0.7.5
boto3                         1.20.11
botocore                      1.23.11
cachetools                    4.2.4
catalogue                     2.0.6
certifi                       2021.10.8
certipy                       0.1.3
cffi                          1.14.6
charset-normalizer            2.0.7
click                         8.0.3
conda-pack                    0.6.0
contextvars                   2.4
coverage                      4.0.3
cryptography                  36.0.0
cupy-cuda100                  9.6.0
cymem                         2.0.6
dataclasses                   0.8
decorator                     5.1.0
defusedxml                    0.7.1
dill                          0.3.4
en-core-web-sm                3.2.0
entrypoints                   0.3
FastNLP                       0.7.0
fastrlock                     0.8
filelock                      3.4.0
google-auth                   2.3.3
google-auth-oauthlib          0.4.6
graphviz                      0.18.2
greenlet                      1.1.2
grpcio                        1.42.0
huggingface-hub               0.1.2
idna                          3.3
immutables                    0.16
importlib-metadata            4.8.2
importlib-resources           5.4.0
ipykernel                     4.9.0
ipython                       5.8.0
ipython-genutils              0.2.0
jieba                         0.42.1
Jinja2                        3.0.3
jmespath                      0.10.0
joblib                        1.1.0
jsonschema                    3.2.0
jupyter-client                7.1.0
jupyter-core                  4.9.1
jupyter-telemetry             0.1.0
jupyterhub                    1.5.0
jupyterlab-pygments           0.1.2
langcodes                     3.3.0
Mako                          1.1.6
Markdown                      3.3.6
MarkupSafe                    2.0.1
mistune                       0.8.4
multiprocess                  0.70.12.2
murmurhash                    1.0.6
nbclient                      0.5.9
nbconvert                     6.0.7
nbformat                      5.1.3
nest-asyncio                  1.5.1
nltk                          3.6.5
notebook                      5.7.11
numpy                         1.19.5
oauthlib                      3.1.1
objgraph                      3.5.0
olefile                       0.46
packaging                     21.3
paddlepaddle-tiny             1.6.1
pamela                        1.0.0
pandas                        1.1.5
pandocfilters                 1.5.0
pathy                         0.6.1
pexpect                       4.8.0
pickleshare                   0.7.5
Pillow                        7.2.0
pip                           21.3.1
preshed                       3.0.6
prettytable                   2.4.0
prometheus-client             0.12.0
prompt-toolkit                1.0.15
protobuf                      3.19.1
ptyprocess                    0.7.0
pyasn1                        0.4.8
pyasn1-modules                0.2.8
pycparser                     2.21
pydantic                      1.8.2
Pygments                      2.10.0
pyOpenSSL                     21.0.0
pyparsing                     3.0.6
pyrouge                       0.1.3
pyrsistent                    0.18.0
python-dateutil               2.8.2
python-json-logger            2.0.2
python-Levenshtein            0.12.2
pythonrouge                   0.2
pytorch-transformers          1.2.0
pytz                          2021.3
PyYAML                        6.0
pyzmq                         22.3.0
regex                         2021.11.10
requests                      2.26.0
requests-oauthlib             1.3.0
rouge                         1.0.1
rouge-metric                  1.0.1
rsa                           4.7.2
ruamel.yaml                   0.17.17
ruamel.yaml.clib              0.2.6
s3transfer                    0.5.0
sacremoses                    0.0.46
Send2Trash                    1.8.0
sentencepiece                 0.1.96
setuptools                    59.2.0
simplegeneric                 0.8.1
six                           1.16.0
smart-open                    5.2.1
spacy                         3.2.0
spacy-legacy                  3.0.8
spacy-loggers                 1.0.1
SQLAlchemy                    1.4.27
srsly                         2.4.2
tensorboard                   2.7.0
tensorboard-data-server       0.6.1
tensorboard-plugin-wit        1.8.0
tensorboardX                  2.4.1
terminado                     0.12.1
testpath                      0.5.0
thinc                         8.0.13
tokenizers                    0.10.3
torch                         1.1.0
torch-tb-profiler             0.3.1
torchvision                   0.3.0
tornado                       6.1
tqdm                          4.62.3
traitlets                     4.3.3
transformers                  4.12.5
typer                         0.4.0
typing-extensions             3.10.0.2
urllib3                       1.26.7
wasabi                        0.8.2
wcwidth                       0.2.5
webencodings                  0.5.1
Werkzeug                      2.0.2
wheel                         0.37.0
zipp                          3.6.0

I wonder how could I solve this bug? Should I use conda or pip to uninstall and install numpy?

@dbast
Copy link
Member

dbast commented Jan 24, 2022

@PolarisRisingWar Can you provide some reproducible commands to reproduce this?

e.g.: `conda install conda-forge::conda-pack && conda create --name testenv packageA packageB && conda-pack --name testenv -o testenv.tar.gz``

Thanks

@PolarisRisingWar
Copy link
Author

I used conda install conda-pack to install this package.
And the codes to reproduce other parts of this environment may be hard, coz it took a long time for me to install and uninstall and update packages in this environment, it's impossible for me to remember every code in this period. I'm sorry for that.

@jeffmelville
Copy link

I'm having a similar issue with Python 3.10 and conda-pack 0.7.0. It listed conda-pack, pip, and wheel as the offending packages. For now I worked around it with --ignore-missing-files. The environment is huge so I'll see if I can narrow anything down. I didn't install anything via pip.

@geoHeil
Copy link

geoHeil commented Aug 1, 2022

I observe this on 3.9 as well

@geoHeil
Copy link

geoHeil commented Aug 1, 2022

it looks like upgrading to the latest version of conda fixes the issue

@github-actions
Copy link

github-actions bot commented Aug 2, 2023

Hi there, thank you for your contribution!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed automatically if no further activity occurs.

If you would like this issue to remain open please:

  1. Verify that you can still reproduce the issue at hand
  2. Comment that the issue is still reproducible and include:
    - What OS and version you reproduced the issue on
    - What steps you followed to reproduce the issue

NOTE: If this issue was closed prematurely, please leave a comment.

Thanks!

@github-actions github-actions bot added the stale [bot] marked as stale due to inactivity label Aug 2, 2023
@mario-dg
Copy link

mario-dg commented Aug 3, 2023

I also have this issue with following yaml file:

name: test_env
channels:
  - pytorch
  - conda-forge
  - defaults
  - anaconda
dependencies:
  - pip=22.1.2
  - python=3.10
  - cudatoolkit=11.6.0
  - numpy=1.22.4
  - pytorch=1.12.1=py3.10_cuda11.6_cudnn8_0
  - torchaudio=0.12.1=py310_cu116
  - torchvision=0.13.1=py310_cu116
  - conda-pack==0.7.0
  - six=1.15.0
  - pillow=9.4.0
  - urllib3=1.26.15
  - pip:
    - PyYAML==6.0
    - grpcio==1.49.1
    - grpcio-tools==1.49.1
    - grpcio-health-checking==1.49.1
    - grpcio-reflection==1.49.1
    - onnx==1.14.0
    - onnx-simplifier>=0.4.13
    - onnxruntime-gpu==1.14.0
    - opencv-python==4.6.0.66
    - pandas==1.3.5
    - pythonnet==3.0.1
    - scikit-learn==1.0.2
    - scipy==1.7.3
    - shapely==1.8.4
    - tensorboard==2.12.0
    - tqdm==4.64.0
    - tensorflow==2.12.0
    - segmentation-models==1.0.1
    - Nuitka==0.6.19
    - numba==0.55.2
    - cupy-cuda11x
    - tf2onnx==1.14.0
    - seaborn==0.12.0
    - matplotlib==3.5.3
    - pystray==0.19.4
    - customtkinter==5.1.2
    - tabulate==0.9.0
    - ultralytics==8.0.145
    - black==23.3.0
    - pytest==7.3.1

Create env with

conda env create -f "environment.yaml"

Then activate the environment and try to pack it

conda activate test_env
conda-pack -p "C:\Users\XXXYourUserXXX\Miniconda3\envs\test_env" -o test_env.tar.gz --n-threads -1 --compress-level 6 --force

When I add the --ignore-missing-files flag, it works, but the resulting .tar.gz is roughly 100 MB smaller.

Edit: I can confirm now, that the smaller .tar.gz is missing files, resulting in a not correctly working application.

Edit 2: If I run following before I try to package the environment it seems to work more reliably.

conda install numpy=1.22.4 -c conda-forge

@github-actions github-actions bot added stale::recovered [bot] recovered after being marked as stale and removed stale [bot] marked as stale due to inactivity labels Aug 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
stale::recovered [bot] recovered after being marked as stale
Projects
Status: No status
Development

No branches or pull requests

5 participants