Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make OpamSwitchState load files lazily (makes lock 4x faster) #194

Draft
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

emillon
Copy link
Collaborator

@emillon emillon commented Aug 6, 2021

Profiling opam monorepo lock reveals that it is spending a lot of time parsing
files (in particular, lexing them) and in OpamPp combinators.
We cannot do a lot about these, but this dominates the profile because opam
parses so many files, tens of thousands of them. It turns out that it parses all
files from all the repositories it knows about.

This PR changes the internals of OpamRepositoryState and OpamSwitchState so
that opam files are parsed only lazily. The name and version of each package is
already known just with the file path, and the contents are often not necessary
at all so they can be hidden behind a lazy.

Some code is left stubbed out (it does not seem to matter to opam-monorepo at
least), but the performance is better: a lock on opam-monorepo itself now takes
1.5s (from 6.5s on main).

@emillon emillon marked this pull request as draft August 6, 2021 14:35
@emillon
Copy link
Collaborator Author

emillon commented Sep 23, 2021

I couldn't reproduce this after upgrading to opam 2.1 but I think that's because my opam2.1 setup used a different OPAMROOT and this setting was ignored.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant