Spec for opam depext integration

This feature aims to integrate the external opam-depext plugin directly into the opam client. This will have the following benefits:

External dependency installation works directly via opam install rather than requiring an additional opam depext subcommand which does not implement the full set of opam install options.
User installations will be faster as the solver no longer needs to be invoked twice (once by depext and once by opam itself)
Better user feedback in the event of an error involving a missing external system dependency.
User choice in the face of multiple external dependencies (MySql vs MariaDB)

Status

Committed for inclusion in opam 2.1.0
Refining the defaults for opam 2.1.0~beta, so the implementation is still subject to change.
@rjbou is adding implementation. Previously, @avsm did first cut, discussed in dev meeting.

Usecases

All of the current usecases of the opam depext plugin need to be supported.

Continuous Integration

For CI, opam depext is usually used as follows:

export OPAMYES=1
opam pin add -n <pkgs>
opam depext <pkgs>
opam install <pkgs> --deps-only
opam exec -- <build>

This sets up the environment such that the local build can be done with a working OCaml and dependency set.

In the case of caching CI systems, there is a slightly more elaborate step:

export OPAMYES=1
# copy *.opam to local build dir
opam pin add -n <pkgs>
opam depext <pkgs>
opam install <pkgs> --deps-only
# copy remaining source tree to local build dir
opam exec -- <build>

This helps preserve caching in the CI so that most of the opam commands are only run if the *.opam metadata changes. An example of a CI that does this is https://github.com/ocurrent/ocaml-ci (which generates Dockerfiles).

The OPAMYES globally ensures that CI jobs do not go interactive. It is also customary to set the CI variable to a non-empty value to indicate that the job is running in a batch scheduler somewhere.

User Installation

Currently many pieces of documentation recommend the following workflow for installing an opam package:

opam update -u
opam depext -i <package>

This is an interactive process; the user is currently prompted to optionally install both the opam-depext plugin, and also from the underlying package manager.

There is also a hybrid use case, to simply record the dependencies needed:

opam depext -l <pkgs>

The result of this can be recorded for later use in a build or package system.

Design Considerations

Security

There is currently a clean separation between opam depext (which can elevate opam to administrator privileges using sudo or doas) and opam install (which runs only with user privileges, and a sandbox for third party code).

This feature combines them both, and so we need to clearly communicate when administrator privileges will be invoked by an opam install.

Using sudo on shared infrastructure such as a university machine will also fail badly. Therefore, the defaults need to be clear that administrator privilege is not required by default.

Performance

Previously, the user could control when the system package manager would be invoked. In some cases (such as Homebrew), this can be quite a slow process. In the case of CI systems, having an ordered set of commands instead of just one helps with caching build results.

Therefore, it is important to have some specific control over when the depext functionality is invoked.

Choice

One example of a tricky package is mysql, since it has the possibility of compiling with either MariaDB or MySQL. Currently, this fails:

$ opam install mysql
The following actions will be performed:
  ∗ install conf-mariadb 2     [required by mysql]
  ∗ install mysql        1.2.4
===== ∗ 2 =====

The following system packages will first need to be installed:
    mariadb-connector-c
Do you want to continue? [Y/n] y

<><> Handling external dependencies <><><><><><><><><><><><><><><><><><><><>  🐫 
Let opam run your package manager to install the required system packages? [Y/n] y
<...>
- ==> Pouring mariadb-connector-c-3.1.7.mojave.bottle.tar.gz
- 🍺  /usr/local/Cellar/mariadb-connector-c/3.1.7: 31 files, 1MB

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><>  🐫 
∗ installed conf-mariadb.2
⬇ retrieved mysql.1.2.4  (https://ygrek.org/p/release/ocaml-mysql/ocaml-mysql-1.2.4.tar.gz)
[ERROR] The compilation of mysql.1.2.4 failed at "make".

#=== ERROR while compiling mysql.1.2.4 ========================================#
# context     2.1.0~alpha | macos/x86_64 | ocaml-base-compiler.4.10.0 | file:///Users/avsm/src/git/ocaml/opam-repository
# path        ~/.opam/4.10.0/.opam-switch/build/mysql.1.2.4
# command     ~/.opam/opam-init/hooks/sandbox.sh build make
# exit-code   2
# env-file    ~/.opam/log/mysql-61660-d2d111.env
# output-file ~/.opam/log/mysql-61660-d2d111.out
### output ###
# [...]
# #define String_val(x) ((const char *) Bp_val(x))
#                       ^
# /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/secure/_string.h:62:27: note: expanded from macro 'memcpy'
#                 __builtin___memcpy_chk (dest, __VA_ARGS__, __darwin_obsz0 (dest))
#                                         ^~~~
# mysql_stubs.c:903:1: warning: missing field 'fixed_length' initializer [-Wmissing-field-initializers]
# };
# ^
# fatal error: too many errors emitted, stopping now [-ferror-limit=]
# 8 warnings and 20 errors generated.
# make[1]: *** [mysql_stubs.o] Error 2
# make: *** [byte-code-library] Error 2

This is because we need to provide a choice of which depexts to install:

  ("conf-mariadb" | "conf-mysql")

similar to Debian's alternatives. The two depexts develop at a different pace.

Implementation

What is added with depexts

on switch load (computation of available packages), a call to sysPM to retrieve available packages status: installed, available, notfound
check that installed packages don't have their sysdep not installed → message + marked for reinstall
on install
- check for sysdep of package to install
- if sysdep not found → fail with not found
- if sysdep available → propose to install sysdep
- if sysdep installed → no op

Scenarios

Generated from master, inluding PR #4194 & #4191.

Simple missing sysdep

$ ./opam install conf-bmake
The following actions will be performed:
∗ install conf-bmake 1.0

The following system packages will first need to be installed:
bmake

<><> Handling external dependencies <><><><><><><><><><><><><><><><><><><><><><>
Let opam run your package manager to install the required system packages? [Y/n] y
+ /usr/bin/sudo "apt-get" "install" "bmake"
[sudo] password for xxx:
- Reading package lists...
- Building dependency tree...
- Reading state information...
- The following NEW packages will be installed:
-   bmake
- 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
[...]

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
∗ installed conf-bmake.1.0
Done.

Missing sysdep of already installed package

$ apt remove bmake
$ opam install ago

<><> Synchronising pinned packages ><><><><><><><><><><><><><><><><><><><><><><>
[ago.0.4] synchronised (no changes)

[WARNING] Opam package conf-bmake.1.0 depends on the following system package that can no longer be found: bmake
The following actions will be performed:
∗ install ago 0.4*

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
∗ installed ago.0.4
Done.

Install package that have a not found sysdep

$ OPAMEDITOR='sed -i -e "s/\[\"bmake/[\"XXX/"' ./opam pin edit conf-bmake
Package conf-bmake is not pinned. Edit as a new pinning to version 1.0? [Y/n] y
Press enter to start "sed -i -e "s/\[\"bmake/[\"XXX/"" (this can be customised by setting EDITOR or OPAMEDITOR)...
[WARNING] The opam file didn't pass validation:
error 53: Mismatching 'extra-files:' field: "detect_program.sh"
Proceed anyway ('no' will re-edit)? [Y/n] y
You can edit this file again with "opam pin edit conf-bmake", export it with "opam show conf-bmake --raw"
conf-bmake is now pinned locally (version 1.0)
[ERROR] Package conf-bmake.1.0 depends on the unavailable system package 'XXX'.You can use `--no-depexts' to attempt installation anyway.
No package build needed.
Nothing to do.

$ ./opam install broken
The following dependencies couldn't be met:
- broken → bsdowl >= 3.0.0 → conf-bmake
depends on the unavailable system package 'XXX', you can use `--assume-depexts' to attempt installation anyway.

No solution found, exiting

Don't let sysPM run by answering no

$ opam install conf-bmake
The following actions will be performed:
∗ install conf-bmake 1.0

The following system packages will first need to be installed:
bmake

<><> Handling external dependencies <><><><><><><><><><><><><><><><><><><><><><>
Let opam run your package manager to install the required system packages? [Y/n] n
[NOTE] Use 'opam option depext-run-installs=false' if you don't want to be prompted again.
This command should get the requirements installed:

    apt-get install bmake

You may now install the packages manually on your system.
When you are done: check again and continue? [Y/n] Y ### NO OP DONE

The following remain to be installed: bmake
When you are done: check again and continue? [Y/n] y ### INSTALLED REQUIRED PACKAGE

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
∗ installed conf-bmake.1.0
Done.

Answer no to all questions

$ opam install conf-bmake
The following actions will be performed:
  ∗ install conf-bmake 1.0

The following system packages will first need to be installed:
    bmake

<><> Handling external dependencies <><><><><><><><><><><><><><><><><><><><><><>
Let opam run your package manager to install the required system packages? [Y/n] n
[NOTE] Use 'opam option depext-run-installs=false' if you don't want to be prompted again.
This command should get the requirements installed:

    apt-get install bmake

You may now install the packages manually on your system.
When you are done: check again and continue? [Y/n] n
Do you want to attempt to proceed anyway? [y/N] n
You can retry with '--assume-depexts' to skip this check, or run 'opam option depext=false' to permanently disable handling of system packages altogether.

Missing sysdep, don't let sysPM run, install anyway. On success, sysdep added to depext-bypass switch option.

$ ./opam option depext-bypass --sw current
[]

$ OPAMEDITOR='sed -i -e "s/.*\/detect.*/[\"echo\" \"FOO\"]/"' ./opam pin edit conf-bmake
Package conf-bmake is not pinned. Edit as a new pinning to version 1.0? [Y/n] y
[WARNING] conf-bmake's opam file has uncommitted changes, using the versioned one
Press enter to start "sed -i -e "s/.*\/detect.*/[\"echo\" \"FOO\"]/"" (this can be customised by setting EDITOR or OPAMEDITOR)...
[WARNING] The opam file didn't pass validation:
    error 53: Mismatching 'extra-files:' field: "detect_program.sh"
Proceed anyway ('no' will re-edit)? [Y/n] y
You can edit this file again with "opam pin edit conf-bmake", export it with "opam show conf-bmake --raw"
conf-bmake is now pinned locally (version 1.0)
The following actions will be performed:
  ∗ install conf-bmake 1.0*

The following system packages will first need to be installed:
    bmake
Do you want to continue? [Y/n] y

<><> Handling external dependencies <><><><><><><><><><><><><><><><><><><><><><>
Let opam run your package manager to install the required system packages? [Y/n] n
[NOTE] Use 'opam option depext-run-installs=false' if you don't want to be prompted again.
This command should get the requirements installed:

    apt-get install bmake

You may now install the packages manually on your system.
When you are done: check again and continue? [Y/n] n
Do you want to attempt to proceed anyway? [y/N] y

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
∗ installed conf-bmake.1.0
[NOTE] Add system package bmake to switch 'depext-bypass' field
Done.

$ ./opam option depext-bypass --sw current
["bmake"]

Same as previous, with --assume-depexts

$ ./opam option depext-bypass --sw current
[]

$ OPAMEDITOR='sed -i -e "s/.*\/detect.*/[\"echo\" \"FOO\"]/"' ./opam pin edit conf-bmake -n
[...]
conf-bmake is now pinned locally (version 1.0)

$ ./opam install conf-bmake --assume-depexts
The following actions will be performed:
  ∗ install conf-bmake 1.0*

The following system packages will first need to be installed:
    bmake

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
∗ installed conf-bmake.1.0
[NOTE] Add system package bmake to switch 'depext-bypass' field
Done.

$ ./opam option depext-bypass --sw current
["bmake"]

./opam remove conf-bmake
The following actions will be performed:
  ⊘ remove conf-bmake 1.0*

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
⊘ removed   conf-bmake.1.0
Done.

$ ./opam install conf-bmake
The following actions will be performed:
  ∗ install conf-bmake 1.0*

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
∗ installed conf-bmake.1.0
Done.

Option depext-cannot-install

$ OPAMEDITOR='sed -i -e "s/.*\/detect.*/[\"echo\" \"FOO\"]/"' ./opam pin edit conf-bmake -n
[...]
conf-bmake is now pinned locally (version 1.0)

$ ./opam option depext-cannot-install ## default
false

$ ./opam option depext-cannot-install=true
Set to 'true' the field depext-cannot-install in global configuration

$ ./opam install conf-bmake
[ERROR] Package conf-bmake depends on the unavailable system package 'bmake'. You can use `--no-depexts' to attempt installation anyway.

$./opam install conf-bmake --no-depexts
The following actions will be performed:
  ∗ install conf-bmake 1.0*

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
∗ installed conf-bmake.1.0
Done.

$ ./opam option depext-bypass --sw current
[]

Option depext-run-installs

$ ./opam option depext-run-installs ## default
true

$ ./opam option depext-run-installs=false
Set to 'false' the field depext-run-installs in global configuration

$ ./opam install conf-bmake
The following actions will be performed:
  ∗ install conf-bmake 1.0

The following system packages will first need to be installed:
    bmake

<><> Handling external dependencies <><><><><><><><><><><><><><><><><><><><><><>
This command should get the requirements installed:

    apt-get install bmake

You may now install the packages on your system.
When you are done: check again and continue? [Y/n] n
Do you want to attempt to proceed anyway? [y/N] n
You can retry with '--assume-depexts' to skip this check, or run 'opam option depext=false' to permanently disable handling of system packages altogether.

Changes from 2.1 alpha to beta

make "no" the default in the prompt.
if "yes" is selected, somehow make the user aware of the other choices?
better discovery of opam option needed somewhere.
opam install --depext-only now exists
removal from the request ...
keep opam depext plugin

Provide feedback

Saved searches

Use saved searches to filter your results more quickly