Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use caching CI builds #753

Open
1 of 3 tasks
mbrandenburger opened this issue Jan 29, 2024 · 10 comments · Fixed by #762
Open
1 of 3 tasks

Use caching CI builds #753

mbrandenburger opened this issue Jan 29, 2024 · 10 comments · Fixed by #762
Labels
feature help wanted Extra attention is needed

Comments

@mbrandenburger
Copy link
Contributor

mbrandenburger commented Jan 29, 2024

The current CI build could be improved to leverage caching. In particular, installing software dependencies (mostly in go) is time consuming and may benefit from caching.

@mbrandenburger mbrandenburger added the help wanted Extra attention is needed label May 6, 2024
@osamamagdy
Copy link
Contributor

Hello @mbrandenburger, can I start working on this?

@mbrandenburger
Copy link
Contributor Author

Hi @osamamagdy, yes please give it a try.

@osamamagdy
Copy link
Contributor

Thanks @mbrandenburger. I started by investigating how to solve the issue. Here is what I found:

  1. To cache any folder with github actions we need to configure a cache key and a path for the artifacts to be cached upon every execution of the pipeline. Normally for projects using golang, it would be $GOPATH/pkg/mod, and if $GOPATH=~/go the caching step would be
    - name: Cache Go dependencies uses: actions/cache@v2 with: path: ~/go/pkg/mod key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}
  2. The most time-consuming step is the run make inside dev container step.
  3. Since the go dependencies installation is running inside a docker container, we can mount a volume to the location where the packages are installed ($GOPATH value inside the container is /project) and cache it on the runner itself -v /project/pkg/mod:~/go/mod/pkg
  4. I locally tried to leave the container running and run go mod download outside and then copy the directory to the container and it didn't take time to install the dependencies

@osamamagdy
Copy link
Contributor

Would this approach affect the running container by any means? like it would fail tests for some reason?

@mbrandenburger
Copy link
Contributor Author

This looks like the right direction to me. 💪

In addition to the go deps, we may also think about caching other artifacts we consume (i.e., fabric binaries, docker images)

@osamamagdy
Copy link
Contributor

osamamagdy commented May 9, 2024

@mbrandenburger Thanks. I am trying to activate the volume mount in the pipeline by assigning a value to DOCKER_DEV_RUN_OPTS before calling the make file like here
I assumed this should work as the makefile is using the ?= operator in here so if the variable is set it should only append to it. but this doesn't seem to work as the make command here is not as expected

@osamamagdy
Copy link
Contributor

osamamagdy commented May 9, 2024

@mbrandenburger It worked by adding a new variable to the make file and appending it to the DOCKER_DEV_RUN_OPTS. Is this acceptable? can you review PR #762 I will revert the changes to the workflow triggers before the changes are merged

@mbrandenburger
Copy link
Contributor Author

@osamamagdy thanks! I will look into this asap!

@mbrandenburger
Copy link
Contributor Author

I re-opened the issue to capture other options for caching.

@mbrandenburger
Copy link
Contributor Author

Thanks @osamamagdy for improving CI caching with the go mod package caching. 🚀💪🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature help wanted Extra attention is needed
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants