conda solves environment very slowly

[xxchan]

Hi, I'm trying to use envd to create a env for this conda env file https://github.com/epfl-ada/2022/blob/main/Exam/adaexam_crossplatform.yml because I have problems to create the conda env directly on my m1 Mac.

def build():
    base(os="ubuntu22.04", language="python3")
    install.conda_packages(
        env_file="adaexam_crossplatform.yml"
    )
    shell("zsh")
    config.jupyter()

But it's very slow:

 => [internal] /opt/conda/bin/conda env update -n envd --file adaexam_crossplatform.yml   2277.4s

It seems these 2 steps are extremely slow:

collecting package metadata (repodata.json)
Solving environment

I also tried creating the conda env directly on a ubuntu machine, and it succeeded in reasonable time.

BTW, there are plenty of results if searching for getting stuck at the above two steps... So it seems a conda issue. Anyway, are there any suggestions about how to make the build faster?

[gaocegege]

I know little about conda thus, ask @kemingy and @VoVAllen for help 😄

[kemingy]

mamba is built for that. Sadly, due to #969 (comment), we are not able to use it.

[VoVAllen]

Hi, I tested the building process on gitpods, and it took about 10min to finish. I think probably the conda resolve depends on the network speed, thus it's very slow on your machine. Since envd is already an isolated environment, I would suggest using pip instead of conda for most of your packages to increase the build speed.

[xxchan]

I think probably the conda resolve depends on the network speed, thus it's very slow on your machine.

I don't think so. I have good network connection :)

Also, 10min still seems long. I just tested on GitHub Codespaces and it takes only 2~3 min to finish conda env create. Especially, "Collecting package metadata & Solving environment" finishes in ~10s, while in envd it takes minites.

[xxchan]

Still think it's a conda issue, and maybe related to the version of conda.. Just curious if there's anything can be done at the envd side, e.g., using a specific version of conda? 🤔 Also as mentioned above mamba seems a good alternative.

[xxchan]

Just curious if there's anything can be done at the envd side, e.g., using a specific version of conda?

I tried to change the conda version used by envd, but it's still very slow to "Collecting package metadata & Solving environment" :(

[kemingy]

@VoVAllen Do you remember the reason why we need to update the conda env instead of creating it from the env.yaml file directly?

[kemingy]

I suspect this strategy also affects the speed since it needs to take the current conda env dependencies during solving.

[xxchan]

FWIW, I just tried conda env create -f using the image continuumio/anaconda3, and solving environments takes only 1s. Quite surprising.

[xxchan]

Even with --platform=linux/x86_64, continuumio/anaconda3 is still much much faster than envd 🤔

[VoVAllen]

I just created an issue for this problem #1409. Thanks for reporting!

[xxchan]

Another mysterious facts:

• If I conda env create after envd up (without the conda env file), it's slow. (i.e., using /opt/conda/bin/conda).

> /opt/conda/bin/conda --version
conda 22.11.1
> conda env create -f ~/ada/adaexam_crossplatform.yml #slow

• If I download another conda, it is fast -- after it finishes, the above one is still resolving dependencies. 😇😇😇

> wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
> bash Miniconda3-latest-Linux-x86_64.sh -b
> /home/envd/miniconda3/bin/conda --version
conda 22.11.1
> /home/envd/miniconda3/bin/conda env create -f ~/ada/adaexam_crossplatform.yml #fast

[kemingy]

Need to consult lawyers.

We don't use the defaults channel ourselves, we only provide the conda environment for users. I don't know if that's allowed by the license.

[gaocegege]

Need to consult lawyers.

We don't use the defaults channel ourselves, we only provide the conda environment for users. I don't know if that's allowed by the license.

Then Let's consult some layers

[gaocegege]

After some legal research and consulting, I think we could use conda defaults.

use solely by an individual using for personal, non-business purposes,
use by a student or employee of an educational institution in connection with educational activities,
use by an employee or volunteer on behalf of a non-profit institution in connection with the provision of charitable services,
use by a 501(c)(3) non-profit research institution in connection with non-commercial activities, such as research to address societal needs and global challenges, public policy development, or the advancement of science for the general good, where such work will not result in revenue to the institution, or use by entities in common control with each other with fewer than 200 employees in aggregate.

anaconda changed its ToS in 2020. use by entities in common control with each other with fewer than 200 employees in aggregate is excluded.

And, we actually do not use it, we just embed the miniconda into the image.

[kemingy]

Actually, we use it if we set the default channel to "defaults". Because when you create the conda env, you need to access the channel service.

[gaocegege]

OK, we use it in a fewer than 200 employees company. Thus it should work