Spec for working with the OCaml compiler

Opam is primarily a tool to manage installation of OCaml packages, but it is also used to manage compilers and different build environments. In opam 2.0 and higher, compilers are exposed as a normal package (so that a package can just depend on ocaml >= 4.08.0), but the OCaml compilers are special citizens of the opam ecosystem. Here are some scenarios we need to support well for opam 2.1:

Usecases

Trying a compiler patch

• “A” works on a compiler feature. They push their changes to a git branch on a fork of the compiler. They call for testers.
• “B” wants to try this patch. They use opam to set up a new switch which is based on this patched version.
  • B can install packages from the main opam repository (packages do not have to know about A’s fork).
  • Some packages can explicitly be available only for this forked compiler.
  • A reference (such as a branch name) is a moving target, so there are ways for B to know which particular commit their switch points to, and to update this reference.

Some example usecases:

• Multicore (https://github.com/ocaml-multicore/ocaml-multicore) is a forked version of the OCaml compiler. This provides a way for users to test it. Library authors can use this to publish packages that depend on multicore features. (see instructions)
• Each PR against ocaml/ocaml is a patched version. This provides a way to test each individual PR. There used to be a GitHub sync that was discontinued due to solver overload when added.

Trying a compiler prerelease

The OCaml developers complete a release cycle. The next version is almost ready. They call for testers.

End users can install this prerelease version through opam. This version is “by default” compatible with packages in the repository, though of course the actual build might fail.

This story is very similar to the previous one, considering the whole release cycle as a large patch. The main difference is that prereleases have stable names (e.g. beta1, beta2) rather than reference names.

An alternative way to look at this is that prereleases are normal compiler versions (like 4.10.0), except that they are not picked by default by the solver. Pypi, npm, rubygems, etc have similar features (this could be applied to packages as well but is out of scope for this story).

Using a locally built compiler to build local projects

Somebody develops a feature on the ocaml repository (compiler or standard library) and wants to use it in a local project. They can set up an opam switch that uses the corresponding compiler. This work does not have to be pushed anywhere. It is possible to install packages in this switch.

Depending on the kind of changes made on the compiler, two workflows are possible to sync the compiler (see https://github.com/gasche/opam-compiler-conf):

• For changes that do not affect the compilation, it is possible to install the compiler in place without recompiling all installed packages.
• In the general case, if compilation changes, it is possible to rebuild all affected packages.

Proposed workflow, from compiler author's perspective:

• Hacking in ocaml, tree is configured and built
• Do something to set-up a (local?) switch (at present opam switch create … and opam pin add --inplace-build)
• Test packages
• Hack on the tree
• Something simple to update the opam view (e.g. opam update)
• Some way to update but not rebuild everything (e.g. for an error message which is updated)

Using a locally installed compiler to build local projects

Somebody builds an ocaml compiler and installs it somewhere (using make install). They can create a switch that uses this install as the compiler. It is possible to install packages in this switch. It is similar to the above workflow, except that it only works on a locally installed compiler, rather than locally built. There might not be a corresponding source tree. Same properties as above apply (it is possible to do a shallow, unsafe update through the equivalent of make install, or a deep, safe update that will rebuild everything).

Feature list

Testing changes on the ocaml compiler

Issue in ocaml: https://github.com/ocaml/opam/issues/4220

(corresponding to "Trying a compiler patch" and "Trying a compiler prelease" above)

• It is possible to create a switch that uses a unreleased version as a compiler
• Packages can be installed in this switch
• Sources
  • Source can be identified by a commit number (e.g. user/ocaml#abcd1234)
  • Source can be identified by release name (e.g. 4.11-rc1)
  • Source can be identified by tag name (e.g. 4.11-rc1)
  • Source can be identified by branch name (e.g. user/ocaml#unbox-int32)
  • Source can be identified by PR number (e.g. PR#1234 or ocaml-multicore PR#1234)
  • Some sources are marked as stable versions and can be selected by default Concretely, creating an opam switch could default to the latest stable version such as 4.10 without being impacted by the existence of a 4.11-rc1 version that compares greater.
  • It is possible to get what stable name corresponds to the moving name of the origin Ex: switch follows user/ocaml#unbox-int32, binaries correspond to commit abcd1234)
• Updates
  • It is possible to change what source a switch uses Ex: move from 4.11-rc1 to 4.11-rc2
  • Moving names can be updated Resolve moving name now and rebuild packages
  • Rebuilds can be bypassed If the change does not modify compilation (messages changed for example), the compiler can be updated in place in an efficient way. This can be unsafe.
  • Installed files can change If the previous state installed the full set of compilers and the new state installs just the bytecode compiler, the previous outdated compilers are removed in the process.

Status

• Under discussion for opam 2.1 inclusion. Not committed yet.
• Feature design in progress.
• Authorship: @emillon @avsm

Implementation