Spec for opam file versioning

Summary

The opam file format includes a statement of its version in the opam-version field. Initially, opam 2.1 included features which would have requiring bumping this to 2.1 in order to enable additional fields. The problems are two-fold:

• opam-repository cannot admit files with opam-version: "2.1", since this prevents any older client from synchronising with the repository
• Source repositories which use opam-version: "2.1" cannot be pinned with opam 2.0, even if there is a meaningful interpretation of the opam file.

The workaround for this is to encode new features using opam-version: "2.0" combined with x-name fields, however this approach runs a risk that the newer fields are never used. This spec proposes an alteration to opam 2.1 to provide a known plugin to be used to downgrade opam files from unsupported versions to allow their being interpreted by opam 2.0.

Status

This specification is being considered for inclusion in opam 2.1 and back-porting to opam 2.0.8.

If approved, it would be worth considering a hardening of the lexing rules for reading the opam-version field - in particular, for opam 2.1+ it may mandated that it's the first field in the file. Changing the lexer (e.g. to add semver) functions shouldn't be done lightly, but at present it's almost impossible.

Implementation

The proposal is to implement the reverse of OpamFormatUpgrade in the opam-state library as a plugin opam-downgrade (so in the package opam-opam-downgrade). The code must necessarily live in a plugin since the intention is that an old client gains the ability to read newer opam files (this is similar to the proposal of moving opam lint back to a plugin to permit newer lint checks to be added to older clients easily).

There are 2½ contexts in which opam would attempt to invoke this plugin:

1. Pin. There are two instances: either no opam file is found at all (because the rules for finding opam files have changed), or the opam file located is for a later version.
2. Update. At least one package in a repository has opam-version for a later version of opam.

For the pin case, the plugin will be invoked receiving pin as an argument and, if it was found, the file opam expected to use. The plugin is executed in opam's copy of the source directory. The plugin is expected to modify the directory. opam will then attempt to repeat the pin operation. This provides considerable flexibility for the plugin - for example, if the location where opam files are to be stored is changed, it may allow the plugin simply to move them to a new location. It also permits emulation of some features (for example, sub-path/sub-packages may generate additional packages for opam 2.0). The plugin is invoked after every update to the repository, before synchronisation with opam's copy (i.e. opam pulls the latest git version/rsync's, runs the plugin and then compares it with the switch's copy).

In the simplest case, the plugin may choose to erase the incompatible fields and use the available field to disable the package. If opam itself could not locate an opam file, and the plugin cannot either, then it may simply do nothing and opam will report a failed pin as it would normally. If opam found an opam file, the plugin must at least convert that file (though it may also write other opam files, and it may erase that opam file).

For the update case, opam, before synchronising the repository contents to the opam root, will identify that at least one opam file has a version it cannot read. The plugin will be invoked with the argument update and in the root of the synchronised repository. The plugin is responsible for scanning the directory and doing whatever is necessary to allow the update to proceed for the client. This flexibility allows the plugin potentially to generate new packages but armed with the context of the entire repository (rather than trying to do that for a single opam file). It also allows for the plugin to be invoked just once per repository, rather than once per opam file (after running the plugin, opam will repeat its normal algorithm for locating an opam file).

In order to limit unnecessary translations for older clients, a lint warning would be added (treated as an error on opam-repository) so that opam-version should be the smallest version necessary to parse the file. i.e. opam-version: "2.1" should only be used if a 2.1-specific field is actually present.

Notes

• This approach allows source repositories and opam-repository to work with latest versions of file formats - in particular, the opam client itself would ever need to use x- fields to interpret current features (the plugin may choose to write x- fields just so that, for example, opam show can display them in an older client).
• It also removes the need for opam-repository or any source repositories ever to carry multiple versions of its opam files.
• opam-repository's CI can still require opam 2.0 installability without preventing the use of newer fields completely.
• In theory, by chaining the downgrade steps in the same way as the upgrader does, it should remain possible to "support" older clients indefinitely, with sunset for a given version being implied only by the widespread use of new features which the plugin translates to "available: false" for older versions of the client.
• The burden of maintaining repository for older clients is shifted to the users (i.e. the people running the client) rather than an opam-repository.