Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use sccache to improve build time #35

Open
dhovart opened this issue Nov 22, 2021 · 12 comments · May be fixed by #81
Open

Use sccache to improve build time #35

dhovart opened this issue Nov 22, 2021 · 12 comments · May be fixed by #81

Comments

@dhovart
Copy link
Contributor

dhovart commented Nov 22, 2021

sccache can be used to cache built dependencies and improve solutions build time.

I may work on this this week.

@dhovart
Copy link
Contributor Author

dhovart commented Dec 2, 2021

@ErikSchierboom I don't know much of your infrastructure.
Is it possible to use if it exists a docker volume for storage that would store:

1/ the compilation cache from sccache
2/ and additionally the rust local registry introduced in #27, that would contain older versions of a crate (instead of regenerating it each time with the crates newest versions as we do each time we build the test runner) ?

@ErikSchierboom
Copy link
Member

@dhovart Currently we do not support external volumes. We might in the future.

@ErikSchierboom
Copy link
Member

@dhovart Couldn't you build an exercise at Docker build time (as in: in the Dockerfile) that uses all dependencies? I imagine that the compilation cache is then populated and that, when running the image, it will re-use that.

@senekor
Copy link
Contributor

senekor commented Jul 18, 2023

This may as a side effect allow people to use compile-time heavy crates, e.g. anything syn-based. Currently, using such dependencies makes the tests time out due to long build time.

@ErikSchierboom
Copy link
Member

This may as a side effect allow people to use compile-time heavy crates, e.g. anything syn-based. Currently, using such dependencies makes the tests time out due to long build time.

Can we precompile those?

@senekor
Copy link
Contributor

senekor commented Jul 18, 2023

I thought that was the point of using sccache, caching the compilation of dependencies. I haven't used it before myself though, so I might be misunderstanding something.

@ErikSchierboom
Copy link
Member

I thought that was the point of using sccache, caching the compilation of dependencies. I haven't used it before myself though, so I might be misunderstanding something.

Oh I don't know much about Rust. What does sccache do exactly?

@senekor
Copy link
Contributor

senekor commented Jul 20, 2023

It caches build artifacts in a more sophisticated manner than the standard toolchain does. C and C++ are supported too.

Currently, every exercise that's submitted is compiled from scratch, including its dependencies. This is both time consuming (costly) and actually prevents some dependencies from being used at all, because they increase build time so much the timeout is triggered.

The proposed solution is to use sccache to precompile the dependencies, embed that cache in the docker image and compile submitted solutions through sccache as well.

I think this would be very beneficial and hope to find time to pick up the stale PR at some point.

(I wonder how much space that cache might take, since we're also talking about reducing the image size atm.)

@dhovart
Copy link
Contributor Author

dhovart commented Jul 20, 2023

Hi, sorry for the PR going stale. Let me know if you'd want me to take this over again.

@senekor
Copy link
Contributor

senekor commented Jul 20, 2023

@dhovart that sounds great! Reading the description of that PR, it kinda depends on this one: #40

I believe the last state of that PR was to try to use dependabot to automatically update version-pinned dependencies. Would you be interested in tackling that?

@ErikSchierboom
Copy link
Member

The proposed solution is to use sccache to precompile the dependencies, embed that cache in the docker image and compile submitted solutions through sccache as well.

That sounds awesome.

@dhovart
Copy link
Contributor Author

dhovart commented Jul 21, 2023

@dhovart that sounds great! Reading the description of that PR, it kinda depends on this one: #40

I believe the last state of that PR was to try to use dependabot to automatically update version-pinned dependencies. Would you be interested in tackling that?

Thanks for your work on this. I actually won't have time to continue working on this one until next week, so feel free to do it if you want to start today!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants