Fix pkgs cache on Windows

[jaimergp]

Description

conda-build's CI uses actions/cache to restore the pkgs cache conda uses to provision the environments. On Unix this takes a few seconds, but on Windows this goes all they way to 6-10mins! Apparently it's caused by tar's slowness (see actions/cache#752).

If this is to save time, I'd say it's not fitting that purpose. If it's to save Anaconda some traffic, then feel free to close.

Checklist - did you ...

• Add a file to the news directory (using the template) for the next release's release notes?
• Add / update necessary tests?
• Add / update outdated documentation?

[jezdez]

It might make more sense to use GNU tar instead to work around the performance issue: actions/cache#752 (comment)

[jezdez]

Actually nevermind, getsentry/sentry-dotnet#1647 seems to indicate it wasn't fixed by switching to gnu tar.

[mbargull]

Ah, I forgot to comment on this one, sorry.
From what it looks like (hard to tell exactly because of fluctuating run times), it kind of cancels each other out.
Meaning, the caching takes far longer on Windows, yes, but actually the package fetching/extracting over the whole test run takes a similar amount of time.
So, to actually reduce the test duration we'd probably really have to fix/workaround the slow cache restoration (e.g., try to pack up the cache and extract manually with some archiver that works faster on Windows). 😞

[jezdez]

Ah, I forgot to comment on this one, sorry. From what it looks like (hard to tell exactly because of fluctuating run times), it kind of cancels each other out. Meaning, the caching takes far longer on Windows, yes, but actually the package fetching/extracting over the whole test run takes a similar amount of time. So, to actually reduce the test duration we'd probably really have to fix/workaround the slow cache restoration (e.g., try to pack up the cache and extract manually with some archiver that works faster on Windows). 😞

https://learn.microsoft.com/en-us/virtualization/community/team-blog/2017/20171219-tar-and-curl-come-to-windows seems to indicate that bsdtar is coming to Windows builds, based on libarchive, which may fix this. We could try to install it from somewhere else?

[jezdez]

I've just applied enableCrossOsArchive to test if this fixes the performance issues. Allegedly that's a feature to create cross-platform compatible caches, but also uses a single faster tar implementation to do that.

[jezdez]

https://github.com/actions/cache/pull/1056/files#diff-f3a6e667d098e7a6f5238b6cc4d54c7c29172b466d5f483840173b377f659a48 is the change related to that

[jezdez]

~6 minutes on Windows :-/

[jezdez]

Aha! When putting conda's package dir on D: instead of C: then it's fast to unpack! Courtesy of actions/setup-go#393

~40 seconds instead of ~6 minutes!

[jezdez]

This is a little nasty, I'll try to upstream this to setup-miniconda

[jaimergp]

Why not use CONDA_PKGS_DIRS env var in the meantime?

[jaimergp]

Ah, it needs to be recorded in the config, I see.

[jezdez]

Ported that to setup-miniconda: conda-incubator/setup-miniconda#328