Update ocaml-base-compiler, ocaml-system and ocaml-variants 4.13.0+ with support for native Windows

[dra27]

This PR augments the three compiler packages (ocaml-base-compiler, ocaml-system and ocaml-variants) with support for the MSVC and mingw-w64 native Windows ports for 4.13.0+. I intend to extend this support back to 4.08 as part of an ongoing overhaul of the compiler packages, but that is beyond the scope of this PR.

Principles

The implementation presented here stems from three underlying principles:

1. All four native Windows ports should be able to be co-installed in the same opam root (i.e. ~/.opam) as separate switches (as with opam-repository-mingw). That is, the user should not be forced to choose permanently between MSVC/mingw-w64 and/or amd64/i686 at opam init.
2. The Windows ports should not be made to look like Tier 2 or alternate platforms; i.e. the instructions to create a Windows OCaml switch should not be fundamentally different from other platforms.
3. Depexts requirements should be precise, so the installation of a conf- package should not speculatively install more dependencies than necessary for the given switch. In particular, if a switch is configured with i686 OCaml, the installation of conf-libfoo should install the required packages for i686 libfoo, not both i686 and x86_64 libfoo.

Corollaries

There are some immediate observations and problems:

• The first principle matches the existing capability of opam-repository-mingw.
• The second principle prohibits either requiring opam exec or needing a Windows-specific wrapper (such as with-dkml or ocaml-env).
• The second principle also has implications for the ocaml-base-compiler package, since there is no concept of a "default" C compiler on Windows. The mingw-w64 and MSVC ports of OCaml are not interchangeable in the way that GCC or Clang-based OCaml is.
• The depext system in OCaml for Windows was based on opam 2.0's opam-depext plugin, and never adopted opam 2.1's integrated support. While the use of the plugin means that it suffers from the same solving problems that caused depext to be integrated directly into the solver in opam 2.1, it did mean that opam depext knew which compiler was installed in the switch, because it was necessarily run after the switch had been created.
• The depexts section of an opam file can only be filtered on global variables. While we can (just about) set switch-specific global variables, this would be both awkward (users would need to know of an extra step), but would also contradict the second principle, adding a requirement to Windows-specific parts of the workflow. This means that the depexts for Windows need to be recorded in separate conf- packages, and the dependency graph of a switch needs to record sufficient information about the architecture and C compiler to select the correct package (in short, we require more base--like packages).

Other related work

This work overlaps with some considerable ongoing additional work on the compiler's opam packages:

• Try a new layout for compiler flags for OCaml 4.12.0 (and add 4.12.0~alpha1) ocaml/opam-repository#17541 added the ocaml-option- packages to reduce the number of variants, but this applies to OCaml 4.12+ only. This facility needs extending (on principle) back to 4.11 and earlier. It makes little sense to update all the individual variants for 4.11 and earlier without first addressing this issue (it's side-stepped for now as 4.13 is the minimum version updated for a different technical reason).
• As I note in Correct the bytecode architectures for OCaml 5.1 and OCaml 5.2 (trunk) ocaml/opam-repository#24075 (comment), Fix ocaml-base-compiler.5.0.0~beta1 on bytecode-only architectures ocaml/opam-repository#22246 forged the wrong approach to fixing bytecode-only 5.x architectures started in Fix architecture support for OCaml 5.x ocaml/opam-repository#21510. This PR doesn't attempt to fix those issues, but the new virtual packages are added in both "input" (ocaml-option-*) form and "output" (base-*), which I believe to be the fundamental error I (/we) failed to see in Fix architecture support for OCaml 5.x ocaml/opam-repository#21510.

I was originally (back in September) of the opinion that it would be better to solve these two issues first and then add Windows support as an extension of these fixes. However, while it's tempting to engineer things such that Windows becomes a "minor" addition, the compatibility concerns for these two fixes make them much higher risk than the Windows packages, which have no compatibility story to worry about. I've therefore restructured the changes so that the alterations are made Windows-only for now, with the fixes to Unix following later, lifting these "limitations".

How it works

With opam 2.2.0 beta2, opam init (if pointed to this branch) will create an OCaml 5.1.1 switch. Concretely, on a clean Windows 11 system:

winget install Git.Git
winget install opam

and then in fresh terminal followed by:

rem Accept all defaults for opam init
opam init git+https://github.com/dra27/opam-repository.git#windows-initial
opam exec -- ocaml

will give OCaml 5.1.1! When creating a switch, an arch- or system- package can simply be added just as for the ocaml-option- packages. For example, assuming the user has installed Visual Studio which, amongst methods, may be done with:

winget install Microsoft.VisualStudio.2022.BuildTools --override "--add Microsoft.VisualStudio.Workload.VCTools --includeRecommended --passive"

then a 32-bit MSVC 4.14.2 switch may be created with:

opam switch create 4.14.2-msvc32 ocaml.4.14.2 system-msvc arch-x86_32
echo print_endline "Hello, world" > hello.ml
opam exec -- ocamlopt -o hello.exe hello.ml

Note that these steps do not require the user to start a Visual Studio Tools Command Prompt or do anything beyond installing Visual Studio.

Under the hood

In more detail, at present, the ocaml.x.y.z package encodes the version of OCaml being installed. To this, at present for Windows only, I have added two more sets of packages:

• arch-x86_32 and arch-x86_64 allow the choice between the i686 and amd64 architectures.
• system-mingw and system-msvc provide the choice between the mingw-w64 and Microsoft Visual Studio (MSVC) ports.

Both ocaml-base-compiler and ocaml-variants use these two sets of packages. ocaml-base-compiler remains "OCaml in its default configuration", but it becomes possible to control exactly which C compiler configuration it's using.

The default compiler is amd64 mingw-w64 (i.e. arch-x86_64 and system-mingw will be automatically added if no other arch- or system- package has been selected) for two reasons:

• We can't detect MSVC using opam's depext system at present (but we can automatically install mingw-w64), so mingw-w64 as a default means that opam init always builds a working OCaml
• Cygwin (and MSYS2) are not available for 32-bit systems, so users will be on 64-bit Windows.

For now, it is intentionally not possible to install the Cygwin port of OCaml using native Windows opam.

Where the user installs a system- and an arch- package, there are also new sets of host-arch- and host-system- which are installed by all opam switches. The idea here is that one host-system- and one host-arch- package are always installed in a switch (be that ocaml-base-compiler, ocaml-variants or ocaml-system).

More information for users

The key rule is that arch- and system- should never be used in opam files because there are packages (such as ocaml-system) which don't use them. These are also packages which may also disappear in the future if opam gains a way to specify configuration options for to packages at installation time.

From the user's perspective, specifying arch-x86_64 vs host-arch-x86_64 is similar to the difference between specifying ocaml-base-compiler.4.14.1 vs ocaml.4.14.1. ocaml-base-compiler.4.14.1 instructs opam to build 4.14.1 from source, where ocaml.4.14.1 permits the use of a system compiler.

The motivation for this change is to be able to indicate precisely where packages are not supported:

• available: os != "win32" is the sledgehammer: no Windows support at all
• conflicts: "host-system-msvc": this package works with the mingw-w64 ports, but doesn't work with the Visual Studio ports
• conflicts: "host-arch-x86_32": this package doesn't work on 32-bit Intel
• depends: "host-system-mingw": this package only works with the mingw-w64 ports.

Notes

I've attempted to organise the changes into a meaningful commit series, which is slightly easier to review than the entire diff in one go. I've added missing metadata fields packages in order to pass opam lint.

TODO

Some final checks, beyond the battery already done on Windows (before this goes to ocaml/opam-repository...)

• Checked Linux installation with this branch!
• Checked rPI 4 (32bit userspace) with this branch!
• Naming decision: host-arch- + host-system- instead of base-arch- + base-system-`?
• Naming decision: ocaml-system-x86_64, etc. or something which is less like ocaml-system itself?

[AltGr]

Incredible work :)

Haven't tested yet but I like the design of it all a lot, it seems pretty sound! Thanks

[dra27]

OK, opened a tracking issue in opam to deal with the 32-bit userspace / 64-bit kernel issue. OCaml builds correctly, but because arch is effectively wrong, we get host-arch-arm64 instead of host-arch-arm32

[dra27]

(but then there was no way before to detect this situation, so nothing is objectively worse)