Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve caching of docker builds #9194

Merged
merged 4 commits into from May 13, 2024
Merged

Conversation

hamishfagg
Copy link
Contributor

This change makes a large portion of the docker build time (pip installing all our requirements) be able to be cached more often.

Cache misses were happening when almost any file changed in the repo, and most requirements would have to be installed again without being cached.

In this PR we instead:

  • create a layer containing only the files required to run pip install . (requirements files, setup.py etc).
  • Copy files from this layer and run pip install . and pip install .[<extras>] (this will now only miss if a requirements file or setup.py etc have changed)
  • Now copy the whole repo into the container and run pip install . again - this will only install MindsDB itself
  • Continue with the build as before

Copy link
Contributor

@mindsdb-devops mindsdb-devops left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see comments, minor layer name missing

Copy link
Contributor

@mindsdb-devops mindsdb-devops left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@hamishfagg hamishfagg merged commit 1d81126 into main May 13, 2024
14 checks passed
@hamishfagg hamishfagg deleted the add/docker_cache_improvements branch May 13, 2024 02:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants