[CI] Enable Conda cache #453
Conversation
|
Reduces time for conda builds for 5 mins. |
|
Created several caches.
So currently it will use 5 caches:
Amount if needed can be reduced with sharing base env and installation on steps. Size of each ~920 Mb (Windows 1.6Gb) |
|
Do we need that many caches? Is it advantageous to cache the L0 deps in their own conda cache vs the speedup we would get by using the general conda cache then install L0 deps? Also, what is the GitHub policy on storage for these caches? |
Devjiu
force-pushed
the
dmitriim/conda_cache
branch
from
d2b656e
to
4b5052d
Compare
Before this pr we used even more conda caches. (about 6 - number of caches = number of keys. I mean we created them but not used) Github cache is limited with 10Gb, So we simply need to fit in it. In common current caches can be found at https://github.com/intel-ai/hdk/actions/caches . Main profit of using so many caches is speed up all jobs. |
|
AFAIU 920 Mb * 4 + 1.6 Gb + 5.6 Gb + 1 Gb ~ 11.8 Gb > 10 Gb So yes, maybe we should have 2 base conda caches: windows and linux. @leshikus WDYT? |
|
During current run - Approaching total cache storage limit (11.62 GB of 10 GB Used) |
.github/workflows/build.yml
Outdated
| @@ -80,6 +80,15 @@ jobs: | |||
| sudo dpkg -i intel-igc-core_1.0.12812.24_amd64.deb intel-level-zero-gpu_1.3.25018.24_amd64.deb libigdgmm12_22.3.0_amd64.deb intel-igc-opencl_1.0.12812.24_amd64.deb | |||
| conda install -n omnisci-dev -c conda-forge level-zero-devel pkg-config | |||
|
|
|||
| - name: Save Conda env cache | |||
| if: steps.conda-cache.outputs.cache-hit != 'true' | |||
There was a problem hiding this comment.
better to name ids for the same steps the same way imho; use either conda-cache or restore-conda-cache, not both in different places
| @@ -16,7 +16,7 @@ jobs: | |||
| - name: Set env context | |||
| run: | | |||
| echo CONDA_PATH=$CONDA >>$GITHUB_ENV | |||
| echo RUN_STAMP=${{ runner.os }}-${{ (env.cuda_compiler_version != 'None') && format('cuda{0}', env.cuda_compiler_version) || 'cpu' }} >>$GITHUB_ENV | |||
| echo RUN_STAMP=${{ runner.os }}-cpu >>$GITHUB_ENV | |||
There was a problem hiding this comment.
both pytest.yml and modin.yml should be made a part of test.yml, thus the whole "Set env context" section will be reusable
There was a problem hiding this comment.
will change it later in further prs
| @@ -44,7 +44,7 @@ jobs: | |||
|
|
|||
| - name: Restore Conda env cache | |||
| id: conda-cache | |||
| uses: actions/cache@v3 | |||
| uses: actions/cache/restore@v3 | |||
There was a problem hiding this comment.
same here, better to have the same action name in all places; if you add "restore" here, add everywhere
This commit enables caching check and unifies cache keys. Signed-off-by: Dmitrii Makarenko <dmitrii.makarenko@intel.com>
Devjiu
force-pushed
the
dmitriim/conda_cache
branch
from
4b5052d
to
799172b
Compare
This commit enables caching check and unifies cache keys.
Signed-off-by: Dmitrii Makarenko dmitrii.makarenko@intel.com