trouble in Azure DevOps test

[yanyew]

I have been experiencing this error continuously during the automatic testing process since submitting the update, but I have not been able to find the cause. It seems that there is an issue with the azure pipelines? It showed "[Errno 28] No space left on device" when TEST START after BUILD SUCCESS
Here is the link to the test in Azure DevOps https://dev.azure.com/bioconda/bioconda-recipes/_build/results?buildId=57320&view=logs&j=e14e69ff-a0ae-55c4-b71d-229b239cfb2f&t=4dddc55b-10bc-50fc-ac68-7ff899b7a030&l=2079

[martin-g]

Can you add df -h before/after the build/test phases ?
Do you download some big data package/file in your tests ?
Which PR is failing with this error ?

[yanyew]

Can you add df -h before/after the build/test phases ? Do you download some big data package/file in your tests ? Which PR is failing with this error ?

I didn't download big data file in the test. And this PR have been completed successfully in the previous PR.
This error occurred this time in https://github.com/bioconda/bioconda-recipes/pull/47492

[TobyBaril]

I am experiencing the same thing with #47498 , and this has also been an issue continuously with RepeatMasker and RepeatModeler too

[martin-g]

Did anyone of you test df -h in the build/test scripts ?

[TobyBaril]

yes, added in the build script, is it worth adding to the test as well?

conda_log.txt

[martin-g]

You can help yourself by adding more df -h in your build.sh and see when the used disk space grows.

E.g. I would bet on https://github.com/bioconda/bioconda-recipes/pull/47498/files#diff-d357f966c7237a539f146074ad4c151fc5fcc6e2514524512aea9a92b6bd692dR20. Here you clone a Git repository. I am pretty sure you don't need all its history, so you can use git clone --depth=1 ...

Once you are ready with it you can delete the local copy of this repo, at line https://github.com/bioconda/bioconda-recipes/pull/47498/files#diff-d357f966c7237a539f146074ad4c151fc5fcc6e2514524512aea9a92b6bd692dR24
Same for https://github.com/bioconda/bioconda-recipes/pull/47498/files#diff-d357f966c7237a539f146074ad4c151fc5fcc6e2514524512aea9a92b6bd692dR59 - I guess you can restore some disk space by deleting the .tar.gz and its uncompressed folder once it is no more needed.

[TobyBaril]

Great thanks, I've made those changes.

I do wonder if this is an issue of RepeatMasker and RepeatModeler dependencies though, as RepeatMasker requires a relatively large database, and the error seems to arise after pulling the container from quay.io. RepeatModeler sources the LTRretriever databases as far as I know - this was also not an issue prior to a small update to the meta.yaml - has something changed recently in the DevOps system?

2024-04-30T09:12:04.3454955Z 09:11:35 �[32mBIOCONDA INFO�[0m (ERR) [Apr 30 09:11:35] DEBU Pull Image [quay.io/bioconda/base-glibc-busybox-bash:3.0]�[0m
2024-04-30T09:12:04.3455747Z 09:11:45 �[32mBIOCONDA INFO�[0m (ERR) [Apr 30 09:11:45] DEBU Packing failed due to open build/dist/conda-meta/font-ttf-ubuntu-0.83-h77eed37_1/licenses/LICENCE.txt: permission denied�[0m
2024-04-30T09:12:04.3456525Z 09:11:47 �[32mBIOCONDA INFO�[0m (ERR) [Apr 30 09:11:47] WARN Local execution errorred with error [open build/dist/conda-meta/font-ttf-ubuntu-0.83-h77eed37_1/licenses/LICENCE.txt: permission denied], retrying with remote execution�[0m
2024-04-30T09:12:04.3457112Z 09:11:47 �[32mBIOCONDA INFO�[0m (ERR) [Apr 30 09:11:47] DEBU Creating container [step-f224588f60]�[0m
2024-04-30T09:12:04.3457921Z 09:11:47 �[32mBIOCONDA INFO�[0m (ERR) [Apr 30 09:11:47] DEBU Image [involucro/tool:latest] not present, pulling it�[0m
2024-04-30T09:12:04.3458380Z 09:11:47 �[32mBIOCONDA INFO�[0m (ERR) [Apr 30 09:11:47] DEBU Pull Image [involucro/tool:latest]�[0m
2024-04-30T09:12:04.3458864Z 09:11:49 �[32mBIOCONDA INFO�[0m (ERR) [Apr 30 09:11:49] DEBU Created container [43bfdb6c19ed step-f224588f60], starting it�[0m
2024-04-30T09:12:04.3459403Z 09:11:49 �[32mBIOCONDA INFO�[0m (ERR) [Apr 30 09:11:49] DEBU Container [43bfdb6c19ed step-f224588f60] started, waiting for completion�[0m
2024-04-30T09:12:04.3460056Z 09:11:49 �[32mBIOCONDA INFO�[0m (ERR) [Apr 30 09:11:49] SERR [Apr 30 09:11:49] DEBU Creating container [step-89d93840f9]�[0m
2024-04-30T09:13:04.1952681Z .09:12:32 �[32mBIOCONDA INFO�[0m (ERR) [Apr 30 09:12:32] SERR [Apr 30 09:12:32] DEBU Packing succeeded�[0m
2024-04-30T09:14:04.1956392Z .09:13:27 �[32mBIOCONDA INFO�[0m (ERR) [Apr 30 09:13:27] SERR [Apr 30 09:13:27] ERRO Task processing failed: API error (500): {"message":"write /usr/local/share/LTR_retriever/database/alluniRefprexp082813: no space left on device"}�[0m

[martin-g]

I am not aware of any changes in the AzureCI config, but there might be something ...
You have added several new dependencies - https://github.com/bioconda/bioconda-recipes/pull/47498/files#diff-c7b20a81f85b48bd9d321f136d14ff451fe825ca2cc1eee9b478a4ade0a2cc35R49-R54 . Maybe they take some space ?!

You can compare the contents of https://anaconda.org/bioconda/earlgrey/4.1.1/download/linux-64/earlgrey-4.1.1-h4ac6f70_0.tar.bz2 > info/about.json against the build logs and see whether something big changed between the builds
Unless you pin a dependency to a specific version, like - mydependency 1.2.3 any following build may use a newer version of this dependency with its new transitive dependencies.

[TobyBaril]

Great, thanks for the information! I'll look into this and see if I can figure something out to successfully merge into Bioconda

[TobyBaril]

Have made the suggested changes and still running into the same issues. Not sure of a way around this to get the new version of the package into bioconda. Currently hosting on my own conda channel in the interim.

https://dev.azure.com/bioconda/bioconda-recipes/_build/results?buildId=57459&view=results

[yanyew]

Did anyone of you test df -h in the build/test scripts ?

Sorry for the late reply. I have test 'df -h' in the test scripts, and found that the disk space occupied reached 97% after the build was completed. Which seems to be wired because the packages used in this build shouldn't occupie so much space.
The link to azure test is https://dev.azure.com/bioconda/bioconda-recipes/_build/results?buildId=57667