Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable APK prefetching in apko_config #134

Open
mattmoor opened this issue Jul 12, 2023 · 0 comments
Open

Enable APK prefetching in apko_config #134

mattmoor opened this issue Jul 12, 2023 · 0 comments

Comments

@mattmoor
Copy link
Member

The basic idea here is to have a provider-level setting (off by default) that directs apko_config to prefetch the APKs to warm the go-apk cache. Something like:

provider "apko" {
  # Have apko_config prefetch the APKs for the relevant architectures.
  prefetch_apks = true
}

This would shift some of the latency from apko_build to apko_config, but we think that practically this will improve throughput because the dependency graph for a common image roughly looks like:

flowchart TB

LC(apko_config.latest) --> LB[apko_build.latest]
LCD(apko_config.latest-dev) --> LBD[apko_build.latest-dev]
LC --> LCD

Today the apko_config rules in the above diagram will be very quick, and the apko_build rules will be scheduled in parallel. Both of these builds overlap in the packages for the latest variant, but if the cache is empty they will likely both try to fill it at the same time and have a very low cache hit rate.

However, if we shift to prefetching the APKs during the config step, since latest-dev depends on latest, we will see latest prefetch all of the shared packages, and only then latest-dev will fetch the things that it additionally needs (likely all hits from other parallel dev builds in a mega-module execution). Then once each builds packages have been prefetched then their builds can be scheduled in parallel without one artificially waiting on the other.

cc @jonjohnsonjr

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant