[ci][docs] updated temp fix for warning written to stderr, fixed conda permissions and updated bench link

[StrikerRUS]

Refer to scikit-learn/scikit-learn#12226.

This temp fix should be removed after scikit-learn 0.20.1 release.

@guolinke Could you please fix dockerfile?

NotWritableError: The current user does not have write permissions to a required path.
path: /usr/local/miniconda/pkgs/cache/3e39a7aa.json
uid: 1001
gid: 1001

If you feel that permissions on this path are set incorrectly, you can manually
change them by executing

$ sudo chown 1001:1001 /usr/local/miniconda/pkgs/cache/3e39a7aa.json

In general, it's not advisable to use 'sudo conda'.

[guolinke]

@StrikerRUS sure, it is building the new docker.
BTW, I am in the vacation. Therefore, I cannot reply to the issues in time.
If there is something urgent, you can email ( guolin.ke@microsoft.com ) it to me.
Thanks!

[StrikerRUS]

@guolinke Thanks a lot!
OK, got it. Have a nice vacation!

[StrikerRUS]

Hmmm, didn't help...

[guolinke]

@StrikerRUS should I add this to docker file ?

sudo chown 1001:1001 /usr/local/miniconda/pkgs/cache/3e39a7aa.json

[StrikerRUS]

@guolinke This won't work because of different name of temp files from run to run.

I've tried different fixes: chmod, chown, unset SUDO_UID. Only sudo chmod -R 777 /usr/local/miniconda works but gives +4 min to build time.

It turned out that this is very widespread problem:
conda/conda#7267
conda/conda#6953
conda/conda#6576

[StrikerRUS]

I suppose that the problem in that CondaEnvironment@1 task is transformed into the following commands

Prepending PATH environment variable with directory: /usr/local/miniconda/bin
[command]/usr/bin/sudo /usr/local/miniconda/bin/conda update --name base conda --yes

executed with sudo.

UPD:
Compare with green master branch:

Prepending PATH environment variable with directory: /usr/local/miniconda/bin
[command]/usr/local/miniconda/bin/conda update --name base conda --yes

[StrikerRUS]

Hi @brcrista ! Can you please help?
A few days ago this task
https://github.com/Microsoft/LightGBM/blob/a694712b7fb86cd532eea2c1781b58d4ba58436a/.vsts-ci.yml#L35-L37
started to use sudo, which produces a well-known conda issue

It turned out that this is very widespread problem:
conda/conda#7267
conda/conda#6953
conda/conda#6576

with NotWriteableError during the next conda usage.

[guolinke]

@StrikerRUS maybe we can update conda by ourself?
I remember there an env-var called "CONDA", which save the conda path.

[StrikerRUS]

@guolinke Yeah! Good point! Let's try this.

[StrikerRUS]

@guolinke Unfortunately, in case of manual conda update another error occurs:

 ERROR conda.core.link:_execute(502): An error occurred while uninstalling package 'defaults::cryptography-2.3.1-py37hc365091_0'.
PermissionError(13, 'Permission denied')
Attempting to roll back.
Rolling back transaction: ...working... done
PermissionError(13, 'Permission denied')

Log: https://lightgbm-ci.visualstudio.com/lightgbm-ci/_build/results?buildId=967&view=logs.

Links: conda/conda#7404, conda/conda#7824, https://stackoverflow.com/questions/49181799/conda-update-conda-permission-error.

Popular solutions are

• chmod/chown which leads to +5 min to build time,
• using sudo which leads to the previous error,
• installation under the same user.

[StrikerRUS]

@guolinke I don't know, but can we solve this issue by changing conda installation folder?
https://stackoverflow.com/a/27364203

[guolinke]

@StrikerRUS
I trigger a new docker build in branch docker, and change the conda location.
you can use tagname ubuntu-14.04-dev to try it.
You also can update the docker branch to trigger the new docker build.

BTW, you can give me your dockerhub account, and I can add you to LightGBM docker org.

Thanks

[StrikerRUS]

@guolinke Thank you very much!

conda update and installation finally work fine with ubuntu-14.04-dev. Please merge docker branch into master and then I'll remove -dev suffix from the docker image here.

Let me register docker account and come back to you.

[guolinke]

@StrikerRUS great! I will create a PR to merge it.

[brcrista]

Thanks @StrikerRUS , I wasn't aware of this problem. I haven't seen it before. Do you have logs you can share? (running with system.debug=true would be most helpful)

I do see the line that says
In general, it's not advisable to use 'sudo conda'.
and found this article: https://conda.io/docs/user-guide/troubleshooting.html#permission-denied-errors-after-using-sudo-conda-command

Right now we have to sudo based on how Conda is installed -- see Microsoft/azure-pipelines-image-generation#259

[StrikerRUS]

@brcrista Thanks a lot for coming here!

The problem is that we use Docker in which Miniconda is installed in /usr/local/miniconda:
https://github.com/Microsoft/LightGBM/blob/a694712b7fb86cd532eea2c1781b58d4ba58436a/.ci/dockers/ubuntu-14.04/Dockerfile#L44-L52
Then CondaEnvironment@1 task runs conda with sudo and after that any attempts to run conda (to be more precise we run conda create)
https://github.com/Microsoft/LightGBM/blob/a694712b7fb86cd532eea2c1781b58d4ba58436a/.ci/test.sh#L11
fail with the error mentioned above.

Here are logs with this error: https://lightgbm-ci.visualstudio.com/lightgbm-ci/_build/results?buildId=945&view=logs

FYI, we've managed to solve this issue by installing miniconda in /opt/conda
https://github.com/Microsoft/LightGBM/blob/cb9043d4ee1ecaf4c43c54930cb3fec6f8c3cfcf/.ci/dockers/ubuntu-14.04/Dockerfile#L44-L54
which allows to avoid using sudo and updating conda by ourselves (without CondaEnvironment@1 task):
https://github.com/Microsoft/LightGBM/blob/2571384e4674a23da580f80bc78cbd4e15b7e8e0/.ci/setup.sh#L60-L61

[StrikerRUS]

@guolinke Here is my docker hub link:
https://hub.docker.com/u/strikerrus/

Thanks!

[brcrista]

I see -- I thought you were on a hosted agent.

For your setup, I would recommend just scripting calls to conda instead of using the task. This usually works better for YAML definitions (the task was made for the old designer workflows).

One difference from using the task: if you're running scripts in multiple steps that operate in the context of the Conda environment, you will need to activate the environment at the beginning of every steps.

[StrikerRUS]

@brcrista Got it! Thank you!