Add a guide/walkthrough #355
Conversation
✅ Deploy Preview for conda-lock ready!
To edit notification comments on pull requests, go to your Netlify site settings. |
|
Thanks a lot for this!!! Quickly skimming I really like the concept. I could see this turning into some sort of quickstart guide. I think people will have a substantial amount of feedback and suggestions, but working through those may be very valuable for bridging the gap between new and experienced users. I am also not sure if/how exactly we will publish this. That is dependent on how this evolves, and also a broader consensus among the maintainers, so I can't make any promises. Also, our review capacity is a bit strained at the moment, so I really hope that spectators step up and offer their feedback. Thanks a lot for all the careful work and thought you have put into this! |
|
You should probably get pre-commit running locally on your side, it is simpler than pytest. |
|
pre-commit.ci autofix |
for more information, see https://pre-commit.ci
|
Make sure to pull that pre-commit formatting fix before making additional commits in order to prevent merge conflicts. |
docs/walkthrough.md
Outdated
| ## Adding a New Package via `conda` | ||
| We have `pandas` installed, but of course we want to use `matplotlib`. | ||
| 1. Install `matplotlib` | ||
| ```bash | ||
| (base) $ conda install -c conda-forge matplotlib | ||
| ... | ||
| ``` | ||
| 2. Add `matplotlib` to the `environment.yml` file. | ||
| ```yml | ||
| # environment.yml | ||
| channels: | ||
| - conda-forge | ||
| - defaults | ||
| dependencies: | ||
| - python | ||
| - pandas | ||
| - matplotlib # NEW | ||
| platforms: | ||
| - linux-64 | ||
| - osx-64 | ||
| - win-64 | ||
| - osx-arm64 | ||
| ``` | ||
| 3. Update the file using `conda-lock` | ||
| ```bash | ||
| (base) $ conda-lock --update matplotlib | ||
| Locking dependencies for ['linux-64', 'osx-64', 'osx-arm64', 'win-64']... | ||
| ... | ||
| ``` | ||
| Note: If you update to a specific version of a package, you will need to provide the version in the command. For example, `matplotlib=3.5` should be included after the `--update` flag. This update flag is useful for the following: | ||
| * Updating to the latest version of a package (no version specified) | ||
| * Updating a package to a specific version | ||
| * Adding a new package |
There was a problem hiding this comment.
Does this actually work? I am trying to do this same type of thing. In my case, I want to add fsspec to https://github.com/firesim/firesim/blob/main/conda-reqs.conda-lock.yml. To do that, I:
- add
fsspecto https://github.com/firesim/firesim/blob/main/conda-reqs.yaml conda-lock -f conda-reqs.yaml -p linux-64 --lockfile conda-reqs.conda-lock.yml --update fsspec
NOTE: I don't necessarily install the package into the conda environment because conda-lock doesn't use the contents of the currently active environment in it's calculation of the lockfile, does it?
However, I find that unless package you're trying to add is in the lockfile, the intersection() at
conda-lock/conda_lock/conda_solver.py
Line 151 in 854fca9
to_update and conda-lock effectively re-solves the entire environment, it doesn't use the update code.
Also, I think it is mildly confusing to suggest step 1 is installing the package into the currently active environment because it implies that having the packages installed somehow influences how conda-lock creates the lockfile, which I don't think is correct. If you are using a lockfile to reproducibly create an environment, it would probably be better to create the updated lockfile and then use it to create the environment. I haven't tried installing on top of an existing environment to see what happens but I'd expect it to be the same as removing the environment completely and reinstalling everything in the lockfile (but perhaps with optimizations for leaving unchanged packages alone).
There was a problem hiding this comment.
The INFO logger line after Locking dependencies for ... is indicative of whether update is happening or not.
When update is being used, it should say something like:
INFO:conda_lock.conda_solver:to_update: ['matplotlib']
I suspect the output looks like :
INFO:conda_lock.conda_solver:linux-64 using specs ['python', 'pandas', 'matplotlib']
and that indicates that the entire environment is being resolved and --update is effectively being ignored.
There was a problem hiding this comment.
@mariusvniekerk, reading #158 it seems like incremental updating should be possible. If you could take a look at this use-case and comment on what I've found, it would be really helpful.
There was a problem hiding this comment.
I can confirm that what is shown in the walkthrough is in fact ..using specs... and appears to be ignoring the --update flag. I can be sure to remove this section, or at least clarify that the environment is being solved from scratch.
There was a problem hiding this comment.
I'm not very familiar with the --update flag, but if there's a suspected bug (or a missing warning message), let's open an issue for follow-up.
There was a problem hiding this comment.
Thanks @mariusvniekerk. I guess I was asking whether it is supposed to be possible to add new specs to the solve and ask to only update those new specs or if one currently must re-solve the whole environment.
There was a problem hiding this comment.
That should work i think, but yeah, probably worth testing if it behaves in the way that you'd expect
There was a problem hiding this comment.
The more standard approach is to just delete and get a new solve
I can modify the document to recommend this approach. However, to your point,
That should work i think, but yeah, probably worth testing if it behaves in the way that you'd expect
It appears the --update flag is not behaving as expected. For example, when I used it to add matplotlib it behaved like this:
(base) $ conda-lock --update matplotlib
INFO:conda_lock.conda_solver:linux-64 using specs ['python', 'pandas', 'matplotlib']This is in contrast to what was expected according to @timsnyder-siv:
INFO:conda_lock.conda_solver:to_update: ['matplotlib']So it appears the command is ignoring the flag.
There was a problem hiding this comment.
@jesshart we should probably open a new issue referencing this. If we're feeling strong in our python-fu, PR'ing a test would also be helpful and if we're feeling particularly masterful, possibly a patch that helps conda-lock do this solve. I looked at the code enough to think I might be able to come up with a patch but it will probably take me longer to get right than I have time for right now. I'll open a ticket from this thread.
docs/walkthrough.md
Outdated
| Sometimes a package installed via `conda` (even using the `conda-forge` channel) breaks my environment. I have found using `pip` as the *exception to the rule* has resolved most of these issues. | ||
|
|
||
| ### Why is this process slow? | ||
| Solving the dependencies of dependencies for one operating system is time consuming. Out of the box, `conda-lock` will try to solve for 3 different operating systems (i.e., linux-64, osx-64, and win-64). However, `conda` recently announced partnering with a `mamba` solver. You could configure your `conda` instance to use a faster solver like this one. This is beyond the scope of this walkthrough. Regardless, it will still take time. |
There was a problem hiding this comment.
conda-lock defaults to using mamba if its installed so we should just recommend that.
There was a problem hiding this comment.
I have been using miniconda with the libmamba solver enabled (ref). Is that much different from using mamba directly?
There was a problem hiding this comment.
they're similar but not the same
There was a problem hiding this comment.
Not trying to be obtuse here, but wouldn't a strong dependency like this discourage a variety of users from engaging conda-lock? I am happy to adapt this documentation, but wanted to approach this philosophical point first.
For example, I could make the point in the documentation so readers are not misled into thinking it only works with the mamba setup.
|
Document Changes
|
|
@maresb I updated the document per the conversations here. However, I forgot to follow your instructions,
Which may have caused a problem. I resolved the merge conflicts on my end, but it looks like they still broke during the |
|
@jesshart, if you like your local version, then the easiest solution is to do a force-push |
|
pre-commit.ci autofix |
for more information, see https://pre-commit.ci
Description
This is the draft walkthrough I mentioned on issue #350.