Add caching to CI test pipeline (#361)

* add-key-with-cache-secret-to-test-task Signed-off-by: Nicolas Höning <nicolas@seita.nl> * correct implementation of the cache action Signed-off-by: Nicolas Höning <nicolas@seita.nl> * indentation Signed-off-by: Nicolas Höning <nicolas@seita.nl> * correct order Signed-off-by: Nicolas Höning <nicolas@seita.nl> * make sure we use the latest stable setuptools Signed-off-by: Nicolas Höning <nicolas@seita.nl> * only install pip-requirements if cache not there. Refactor SETUP script, to make it only about postgres installation Signed-off-by: Nicolas Höning <nicolas@seita.nl> * install pip-tools, no use for secret for now Signed-off-by: Nicolas Höning <nicolas@seita.nl> * cache the entire Python environment Signed-off-by: Nicolas Höning <nicolas@seita.nl> * better name for postgres script Signed-off-by: Nicolas Höning <nicolas@seita.nl> * changelog entry Signed-off-by: Nicolas Höning <nicolas@seita.nl> * add possibility to invalidate cache if needed Signed-off-by: Nicolas Höning <nicolas@seita.nl>
FlexMeasures · Apr 15, 2022 · 9207b15 · 9207b15
1 parent b2d16c3
commit 9207b15
Show file tree

Hide file tree

Showing 6 changed files with 26 additions and 8 deletions.
diff --git a/.github/workflows/lint-and-test.yml b/.github/workflows/lint-and-test.yml
@@ -34,9 +34,21 @@ jobs:
         run: |
           git fetch --prune --unshallow
           git fetch --depth=1 origin +refs/tags/*:refs/tags/*
-      - run: ci/SETUP.sh
+      - uses: actions/cache@v2
+        id: cache
+        with:
+          path: ${{ env.pythonLocation }}
+          # manually disable a cache if needed by (re)setting CACHE_DATE
+          key: ${{ runner.os }}-pip-${{ env.pythonLocation }}-${{ SECRETS.CACHE_DATE }}-${{ hashFiles('**/requirements/**.txt') }}
+          restore-keys: |
+            ${{ runner.os }}-pip-
+      - run: |
+          ci/setup-postgres.sh
+          sudo apt-get -y install coinor-cbc
       - name: Install FlexMeasures & dependencies for tests
+        if: steps.cache.outputs.cache-hit != 'true'
         run: |
+          make install-pip-tools
           make install-for-test
           pip install coveralls
       - name: Run all tests except those marked to be skipped by GitHub

diff --git a/ci/Readme.md b/ci/Readme.md
@@ -1 +1,4 @@
 # Continuous integration
+
+Here are some useful script for CI.
+
diff --git a/ci/install-cbc.sh → ci/install-cbc-from-source.sh b/ci/install-cbc.sh → ci/install-cbc-from-source.sh
@@ -1,5 +1,10 @@
 #!/bin/bash
 
+######################################################################
+# This script installs the Cbc solver from source
+# (for cases where you can't install the coinor-cbc package)
+######################################################################
+
 # Install to this dir
 SOFTWARE_DIR=/home/seita/software
 if [ "$1" != "" ]; then

diff --git a/ci/SETUP.sh → ci/setup-postgres.sh b/ci/SETUP.sh → ci/setup-postgres.sh
@@ -1,19 +1,16 @@
 #!/bin/bash
 
 ######################################################################
-# This script sets up a new FlexMeasures instance in a CI environment
+# This script sets up a new Postgres instance in a CI environment
 ######################################################################
 
 
 # Install dependencies
-apt-get update
-sudo apt-get -y install postgresql-client coinor-cbc
-make install-deps
-
+sudo apt-get update
+sudo apt-get -y install postgresql-client
 
 # Wait for the DB service to be up.
 
-# Hack until this feature is ready: https://bitbucket.org/site/master/issues/15244/build-execution-should-wait-until-all
 statusFile=/tmp/postgres-status
 while [[ true ]]; do
   telnet $PGHOST $PGPORT &> ${statusFile}

diff --git a/documentation/changelog.rst b/documentation/changelog.rst
@@ -16,6 +16,7 @@ Bugfixes
 Infrastructure / Support
 ----------------------
 * Unit conversion prefers shorter units in general [see `PR #415 <http://www.github.com/FlexMeasures/flexmeasures/pull/415>`_]
+* Shorter CI builds in Github Actions by caching Python environment [see `PR #361 <http://www.github.com/FlexMeasures/flexmeasures/pull/361>`_]
 
 
 v0.9.3 | April 15, 2022

diff --git a/documentation/tut/installation.rst b/documentation/tut/installation.rst
@@ -235,7 +235,7 @@ Installing Cbc can be done on Unix via:
 
 (also available in different popular package managers).
 
-We provide a script for installing from source (without requiring ``sudo`` rights) in :ref:`continuous_integration`.
+We provide a script for installing from source (without requiring ``sudo`` rights) in the `ci` folder.
 
 More information (e.g. for installing on Windows) on `the Cbc website <https://projects.coin-or.org/Cbc>`_.