Enable specification of conda dependencies for apps

[danielhollas]

While I understand the new paragraph in README.md about the special volume that AiiDAlab launch creates for local conda environments, I am still unsure how to use it in practice.

To be concrete, in my case I need to install xtb-python as a dependency of my AiiDALab application. This package is not published on pip and must be installed via conda-forge channel.

So far I've been installing it as root

aiidalab-launch exec --privileged -- conda install xtb-python

If I wanted to install as aiida user, my understanding is that I'd need to create a new conda environment. Should this new environment be a clone of the base, like this?

conda create --name xtb --clone base

And then I guess I need to automatically activate it, perhaps in /home/aiida/.bashrc? What is the best practice here?

Thanks!

[csadorf]

Sorry for not getting back to you earlier. I was on vacation for a few days and was still in catch-up mode.

Indeed, the base environment cannot be changed by a non-root user. However, you can always create a separate environment for the specific needs that you have and also use it as a Kernel for your notebooks and apps. I don't think we really have a standard approach for this scenario.

The main challenge of simply putting everything into separate environments is that aiida-core needs to be installed in a consistent location and also expects other dependencies to be available there.

[danielhollas]

@csadorf no worries, thanks! Would you happen to have a working example somewhere of setting up and using nondefault conda environment for the AiiDAlab apps? I looked into it a bit and it seems nontrivial. What I would really like is to make so that when I do aiidalab-launch start the app uses the new environment automatically (without e.g. switching the kernel).

[csadorf]

@csadorf no worries, thanks! Would you happen to have a working example somewhere of setting up and using nondefault conda environment for the AiiDAlab apps? I looked into it a bit and it seems nontrivial. What I would really like is to make so that when I do aiidalab-launch start the app uses the new environment automatically (without e.g. switching the kernel).

No, I don't think such an example exist yet. Just to make sure that I understand correctly: Your primary motivation is to install additional dependencies that are required by an app that cannot be installed by pip, right? The replacement of the default conda environment is not the primary motive, or is it?

[danielhollas]

Yes, exactly, I need additional dependencies via pip, and want them to persist between the AiiDAlab launch invocations.

[csadorf]

Yes, exactly, I need additional dependencies via pip, and want them to persist between the AiiDAlab launch invocations.

I thought they cannot be installed via pip?

[danielhollas]

Uh, apologies, I meant via conda.

[csadorf]

Ok, I do not have a straight-forward solution for this at the moment. However, I am going to put it on our agenda to discuss.

[danielhollas]

Update on this.

The original issue here was that one could not put the conda install in the post_install script, because that script was run under aiida user, but installing into conda environment required root privileges. However, I think in the new Docker stack images that is no longer the case since the conda environment is managed by the jovyan user. So this issue might be solved already. I will test whether conda install works within post_install. If it works, the only thing to be done is here is to document this somewhere (but I think the AiiDAlab documentation needs a bigger overhaul and update anyway).

I have reduced the urgency to 'low' because it's no longer a blocker for me since I am building my own Docker image anyway. If anybody else runs into this then it can be increased again.

cc @unkcpz @yakutovicha

[danielhollas]

@unkcpz @yakutovicha this issue in on the wrong repo. Would you mind moving it either to the docker-stack or aiidalab repositories?

This is related to our discussion in aiidalab/aiidalab-docker-stack#343