Skip to content

2021 Developer Meetings

Jump to bottom
David Allsopp edited this page Jan 7, 2022 · 63 revisions

17-Dec-2021

Last week

  • David: glibc 2.34 back-ports for ocaml in opam-repository. Also working on GHA Windows - mingw-w64 working to the point that the error is actually a "real" fault. Cygwin needs an irritating, but obvious tweak, and msvc64 needs an OCaml fix to be forward-ported.
  • Louis: Exiting the rabbit hole gradually. PR opened to change orphan-handling! Little bit of extra work needed to handle "post"-dependencies, but it's working nicely. Ported a lot of the old tests from opam-rt to the reftests framework which exposed a few weirdnesses. Would like to finish this off with opam pin add --current which allows you to recover an orphaned package by promoting the orphaned package description to a pin. Once that's done, will be back to --deps-only
  • Raja: many tests and fixes! Tests for opam lock and pinning lockfiles which found some bugs (which got fixes). Tests added for all linting steps - which also revealed some unreachable linting errors. Fixes to archive checking - one was checking extension-only and the other part checking the file; both now check the file is it exists. opam repo set-url with a fetch-failure would leave the repository in an empty state (whereas opam repo add reverts adding the repository).
  • Kate: opam exec --revert-env PR; fix OpamSystem.real_path on newer systems (@dra27 to check what's going on with Cygwin). Looking into issue with main instance of opam-health-check crashing - suspicious initially of opam libraries, but they're now vindicated!

Notes

  • Release plan for 2.2 drafted (@dra27 to publish to wiki)
  • Various PR discussions (notes on PRs)

3-Dec-2021

Last week

  • Kate: PR for opam-0install-cudf for supporting the avoid-version flag - implemented by looking if there's an avoid-version flag. It does a double-solve if an avoid-version package was detected. Tweaked opam state to check for updates more frequently (ocaml/opam#4935).
  • Raja: Versioning pinning questions for archives. Related to that, found some bugs in pinning when using lockfiles (starting from ocaml/opam#4936). Currently working on a change to the reftest driver for handling URLs in opam files for testing (not yet pushed)
  • Louis: Progress on install --formula and re-implementing --deps-only and working out handling for corner-cases. Working on tests for other corner-cases arising from going from a list of atoms to a general formula. Found some quirky code in --assume-build which may want cleaning up: it's only referenced in OCaml's hacking documentation. Did some experiments with git-lfs and managed to have a branch on OCamlPro's GitLab clone of opam-repository which includes all the tarballs.

Notes

  • Software Heritage: @AltGr reported that it's moving along nicely. Library in OCaml for generating the hashes and handling the URLs. All looking on track for 2.2
  • 2.1.2: #4908 is ready to go in and then back-port to 2.1
  • Camelus: @AltGr: Camelus has no longer been maintained, is this worth trying to resurrect? @dra27 - it'd be nicer to be putting the actual checks into opam-repo-ci and then separate out surfacing the result into the PR (i.e. commenting instead of having it with logs). @kit-ty-kate - opam admin check --installability is available which could be dispatched as a build job and parse the logs. @AltGr - this could possibly be extended to allow the dependency cone to be limited for performance? @AltGr - it's possible that the old installability check was run too often before (for example, it should possibly only be run for changed packages). At present, opam-repo-ci clearly handles when a single package is not installable at all (since the tests will fail overall) but if multiple packages are in a single PR and only one of them isn't installable (or at least one of them is) then you have to remember to check the logs.
  • Question from Khady regarding unexpectedly unconfigured environment (ssh from within VS Code): this harks back to lots of previous issues on the subject. @dra27 - why don't we improve the env sync warning at this point, for when we're non-interactive AND there's nothing in the environment to suggest that it may be that your shell is not configured.

26-Nov-2021

Last week

Raja: Installed opam on Windows! Allowed a failing test to be identified - problem of escaping backslashes. The other problem is the old chestnut of double-printing of commands. Going to be having a look at recpin on Windows next. Added PRs for other 2.2 issues. Louis: Continued a little on opam install --formula. Quite a lot of questions still to resolve on exactly how the feature is equivalent (for example, there's a lot of processing done on a list of atoms and it's not clear if those should be applied when the list is part of a formula). For example, there's automatic update of pinned packages which are specified on the command line, but now they could be part of an or branch, for example. Kate: opam-build and opam-test plugins. Becoming more familiar with the opam API. Under the hood the two plugins are doing similar things, so the aim now is a third plugin (opam check-constraints) which would:

  • Create a local switch (if not present)
  • Install the package with most recent versions of everything
  • Install the package with most recent versions + tests (and run them)
  • Repeat those two steps with lower bounds This is blocked at a failure at the moment with an inconsistency in opam-client.

Notes

4932

XXX

4925

Two possible issues:

  • It's not clear from the opam display that the package is pinned
  • Could we add a version?

We could change the * to be something clearer like " (pinned)" and then also consider adding a suffix to the version itself (with the potential for breakage with that being that you have to either specify the version directly on the pin command or have a version field in the opam file in the repo).

4928

@kit-ty-kate - xref'ing https://github.com/ocaml/opam/issues/4647 will of course reduce these reinstalls. --skip-reinstalls @AltGr - some installations (e.g. from file-depends) are "mandatory" @dra27 - although even then do they have to be?

Notes

Kate's plugin work - are the caches working side-by-side e.g. opam-dune-lint with 2.0.9 library and 2.1.0 switch? @dra27 will check - it sounds like something isn't working correctly with the side-by-side repo caches.

PR 4933

Could possibly add a flag internally so that we can identify the first opam update since the url was set so that the warning could be better (to differentiate, say, the network being down and not warning about all the URLs).

19-Nov-2021

Last week

Louis: Adding ability for dependency requests into the solver. At the moment, the invariant is only possible as a list of packages. Motivation for this is the --deps-only feature and the number of issues which are affected by it. The present implementation works by solving the packages and then filtering out the packages themselves - new approach is instead to ask the solver to solve the dependency list directly. Discussion on this feature: @dra27: could we introduce this experimentally to start with, just to ensure there's an escape hatch if issues are found (and then aim to switch over in a release or two's time). Raja: Working on the bug-fix version of the --deps-only fix. New option for opam clean --untracked. Eliminated polymorphic compare in various places (at the moment for hygiene, rather than necessarily for performance) - underlying problem was wanting directory comparison. Kate: Various PR reviews; new PR for making 0install solver mandatory (just waiting for review); new PR to implement opam switch -.

Notes

  • @kit-ty-kate had a discussion with @c-cube on opam pain-points. Issue with lock-files: opam-repository can end up breaking existing lockfiles because dependencies break. In this case, the problem had happened because a package had been edited.

12-Nov-2021

Last week

Louis: Nothing to report on a quiet week David: Working with Raja on the blog post; having a look at the posix sh sandbox rewrite Kate: Lots of PR reviewing. Started working on the branch to bring in 0install solver as a full dependency. Raja: Sweeping through issues to look at for 2.2. 2.0.10/2.1.1/1.2.0 releases. Had a look at adding configuration information to the debug logging - slight recursive problem because the configuration is initialised higher up than the debugging. PR rebasings.

Loose-ish agenda:

  • PackagingCon
  • opam.ocaml.org
  • Blog post
  • Status of #4900
  • Question from setup-ocaml - should we make --assume-depext the default in CLI 2.0? - @dra27 do a quick PR?

Notes

Faster releases:

  • Use a staging repo for the tag so that one person can set-up the tag and kick off the builds so that the script is done in one place and the tag and release binaries are pushed at once.
  • Signing key needs to be shared with a couple of people
  • Replicating the VMs

Discussions around 4891 -> 4831 -> 4781 -> 4419

  • @dra27: do we possibly need to look at ways of syncing the .changes files (so update the time-stamps to reflect what's there)
  • @kit-ty-kate: we could in these situations move the files/directories to a save place and then the warning can be that the files were saved

5-Nov-2021

Slightly chaotic notes from @dra27!

Last week

  • Raja: debugging refactored parallel fetch... almost finished. Doing some more testing - expecting PR later today. Discussions on #4885 - looking at alternatives to #4886. 2.0.10 and 2.1.1 tagged!
  • David - opam-depext review + opam-sources

Notes

  • Release process
    • Perhaps @AltGr should move towards having Dockerfiles and scripts to generate the release artefacts?
    • @dra27 - can we trust GHA to do the builds? @AltGr had a wip branch (literally) on ocaml/opam from June which could be continued.
  • opam-version downgrade - with a sync'd repo for strictly < 2.0.11 (or whatever) and between (2.1.0 and whatever). Idea here would be to include the simplest possible in a 2.0.x and 2.1.x and 2.2.x release that opam-version not recognised is simply translated to available: false. We can maintain a redirected repository for the older opam releases (at least for the default http remotes) because in this case the rewrite is completely mechanical (no need to be using opam itself to upgrade the file, etc.)

29 Oct 2021

Last week

  • Louis: is back! Looking at solver-relating things. Big improvement to the Z3 solver (including a correctness fix). Need to double-check if the correctness bug is in 2.1 - it might be tricky to separate it from the performance improvement. At the moment of course the major use-cases of Z3 have switched to 0Install, so we're not seeing as much continuous testing of it. The Z3 solver has the big advantage over the other solvers that it can abort with a non-optimal solution instead of timing out, so it remains worth maintaining in case we hit the wall again! Discussion on 4880 - @dra27 quite keen that we could look at parallelising the checks - we can terminate the Dose thread easily and default mccs can be terminated and just ignore the thread otherwise. @kit-ty-kate - can also do both; so we have the control of the preemptive check and then have parallel version on top.
  • Raja: working on some fixes for the working directory checkout corner cases and for checking the sandbox (4879). Spotted while testing that the macOS sandbox is able to write to the global root configuration during an install, but the Linux sandbox. Also cleaning up the code for parallel fetch. Subpath pinning and opam-file-format - recalling the concern that even though the syntax is meant to be used internally that it being there might mean that people want to use it in repos.
  • Kate: Less eventful opam week - PR on improving the interaction of opam init
  • David: 4841 rebased on to 2.0.10. Working on glibc 2.34+

Agenda

22 Oct 2021

dra27 failed to take notes

15 Oct 2021

Short update

  • Raja: mainly CI related. Looking at the depexts GHA workflow. Updating the subpath pinning spec and ensuring the reftests are up-to-date. Working with Leo on the paper for Software Heritage work. Also dealing with opam.ocaml.org VM issues.
  • David: OCaml Cafe talk - being reminded of all the nice things added in opam 2.1 🙂
  • Louis: not too much opam. Helping with adding software heritage information to opam files. A fake URL feels like the best way of doing this.
  • Etienne: some work on opam-monorepo and CI, but nothing on opam specifically.

Last week

8 Oct 2021

Agenda

  • PackagingCon 2021
  • OCaml Café Introduction to opam
  • OCaml Users Survey / opam survey
  • Issue: #4856 - certificate issue, so close?
  • Issue: #4860 - opam-repo-ci CI for depexts?
  • New opam.ocaml.org VM status
  • opam 2.0.10 / 2.1.1 status
    • #4841 - good to go?
    • #4853 - tweak to message, but good to go?
    • #4861 - agree on where to place test; @dra27 to triple-check that it actually works on Windows again, but otherwise good to go?

Last week

  • Raja: depext work; working with @zapashcanon on Software Heritage: simplified some of what's being proposed, with some PRs now open. For now, it will be a fallback mechanism only and it can be very slow at the moment.
  • Louis: mainly learn-ocaml this week, but also reviewing opam PRs.
  • David: rewrote the hacked revert-env fix and opened a PR for it (mainly on other things too)

Notes

  • OCaml Café - demonstrate switch creation improvements (can mention opam update speed as well) in particular upgrading compilers, integrated depext (MariaDB/MySQL) and lockfiles. Can also demonstrate fast switch creation from the relocation work and discuss other new features planned for 2.2.
  • OCaml Survey - let's aim for three questions to be added to the main one and then use the feedback from that (possibly if we can have a "Would you fill in a full opam survey (yes/no)" too!)
    • A check-all-that-apply question for use of top-level commands (i.e. "I use opam for:...")
    • Global vs Local switches (might be a radio + textbox or perhaps a radio + some checkboxes, e.g. "I mainly use: " and "I have at least once: " or even a matrix of radio options (Never/Sometimes/Often, etc.)
    • Usage of pinning (check all that apply: version pins, pin-depends, pinning local sources, pinning remote sources, etc.)
  • Bigger survey might be possible in conjunction with other tools?
  • opam-dev.ocaml.org - @dra27 will see where we are with upgrading the VM next week
  • opam.ocaml.org - @AltGr and @rjbou will sort out a cron job for backing up the cache (as it's rapidly becoming an archive rather than a cache!)

1 Oct 2021

Last week

  • David: translating relocatable compiler demo to Windows revealed a couple more little things which are worth looking at: there's a long-standing bug with environment updates which themselves include multiple items (i.e. contain an embedded colon or semicolon) - they don't revert. It's also worth adding some detection with local switches to regenerate the environment file if the switch's directory has changed.
  • Raja: first cut of de-duplicating fetches done. Still various things to be done to it (code duplication); opened #4853 with the CLI versioning fix for 2.1.1. Also discussions with Léo this week on the PRs for changes required for Software Heritage.
  • Louis: no opam this week

2.1.1

  • @dra27 opened #4854 to track updating opam-depext in sync with the 2.1.1 release; the aim is that OPAMCLI=2.0 opam depext should be able to work as expected to install depexts as inefficiently as in 2.0, but to allow CI systems to use it.
  • #4841 needs merging and then cherry-pick to 2.0.10 and 2.1.1
  • @dra27 will spin the opam env --revert fix into a PR for 2.0.10/2.1.1 as well (same basis as #4841 - it's probably the underlying cause of previous strange bug reports!)
  • @AltGr will review #4853 - discussed various levels of deprecation (keeping --packages in 2.1+ CLI). --packages is a useful idiom for users (and scripts) which don't need the full power of --formula (which has complex escaping requirements, especially in scripts) but being able to mix --packages and a list of normal positional parameters is just unclear. So, as in the PR, that mode will be kept for the 2.0 CLI only, and the error message will remain if both packages and --packages are specified in the default CLI (and 2.1+)

Global formulae

  • Discussion around generalising switch invariants further. The motivation would be to allow users to specify packages to be installed in switches by default.
  • opam init would by default leave this invariant empty - it would be advertised as a useful feature, for example, to have merlin and ocamlformat automatically added to new switches. The invariant would be added to the switch invariant.
  • From the user's perspective, it's useful to have the same --formula and --packages modes as for opam switch create - most users will want to specify a list of default packages.
  • @rjbou: installing the default invariant as best-effort (before fixing the invariant, therefore) would allow having the tools you want listed, but still be able to create switches where some of those tools can't be installed
  • @dra27: could also look at additional opam switch commands for updating the invariant: e.g. something like opam switch add-invariant to append additional packages to the invariant, rather than having to re-write the entire invariant (e.g. opam switch add-invariant merlin)

24 Sep 2021

There was no meeting

17 Sep 2021

Last week

  • David: relocatable compiler demo uncovered two little bugs in opam. #4326 wasn't cherry-picked from the 2.0 branch (fix included in 2.0.8) so is missing in 2.1 (see #4840) and a bug with setenv when a package is installed (see #4841).
  • Louis: just back from holidays, and been on OCaml this week.
  • Raja: working on deduplicating fetches (#3741 and #4056) and debugging sandboxing issues on macOS (see #4797 (comment)). Looking at a bug Fabrice hit with working directory paths - rsync failing because an absolute path had been prefixed with the working directory. Discussions with Léo (@zapashcanon) over updating source links to use Software Heritage as a backend.

Issues & PRs

  • #4841 - @dra27 proposes putting this in 2.0.10 as this is probably the cause of some weird bug reports in the past relating to stublibs etc. and especially affects first time users of OCaml (who may have init'd opam but not yet installed a package).
  • #4830 - @AltGr: the original idea here is that the repository cache was likely to be better if you have more than one repository configured (e.g. Coq). The only downside to the change is the extra round-trip to miss the global cache if it doesn't have it. @dra27: wasn't the original use-case here to avoid network access - could we instead consider having a setting which disables repository caches or which adds a global override cache? @AltGr: there is the related quirk that git remotes don't use the cache configuration. @dra27: is that the same for local remotes? Action: @dra27 will check 2020 notes and see what else was discussed.
  • #4838 - the first part is a feature, it's a legacy of OPAM 1.2.2 that only one repo and one package can be given (it made no sense to specify multiple repositories in OPAM 1.2.2 when you could only install a single compiler description as part of opam init); @dra27 will open an issue to track the second part, which is a CLI versioning bug and which we should squash for 2.1.1.

10 Sep 2021

Agenda

  • opam.ocaml.org server
    • opam2web had updates, but wasn't compiling on the aged opam.ocaml.org VM
    • opam was updated, but the opam2web binary wasn't built against 2.0.9, so wasn't loading the state
    • Manually downgraded the root on the VM and some other cleaning up
    • Looks like @avsm has the new machine working - but is it deployed?
    • @dra27 - worth adding a firehose for updates, e.g. a slack channel? @emillon's - could also use Dead Man's Snith
  • opam 2.1.1
    • Waiting on #4833
    • Waiting on #4791 - @dra27 will check the AppVeyor failure
  • Issues
    • #4828 - any ideas how this is triggering? _
    • #4829 - feasible feature? Yes - added to Features project
    • #4832 - additional linting? Improve the warning?
    • #3741 - shared fetch? @rjbou started working on it.
  • Other discussions
    • opam-test plugin and #4594

3 Sep 2021

Agenda

  • PackagingCon 2021 submission
  • External PRs
    • Volunteer to rebase and merge #4039
    • Split off the uncontested part of #4421 and try to conclude the rest of the discussion on the remain bit?
    • Reviewer for #4803 - @AltGr?
    • Conclusion for #4804 - further changes for @AltGr?
  • Issues
    • #4809 - opam source working in a bare opam root
  • Specs
    • Shell and Sandbox updates (@dra27)
      • Obviates #4414 and #4816. This change should make dealing with changes in the external environment for opam easier to deal with. If accepted, it's worth using as the Windows shells go in for 2.2

Notes on the spec

  • opam update -u => should not by default update the scripts
  • opam upgrade => the question for updating should not default to yes - so a single question if interactive but with --yes the sandbox upgrade question should be separate and default to no
  • local switches - less usual to run opam upgrade here, so may want opam switch create to advise that the scripts are out of date (in the same way as the opam repositories not having been updated is warned)
  • @rjbou - further warning could be for any action which writes the switch state to warn that the scripts are out of date. For example, allowing opam install wants to check that the sandbox is up-to-date too.
  • @emillon - what about user's who don't want to ugprade the sandbox.
  • Can change it so that the sandbox is never automatically installed, but there's always a warning (and a global option to control its installation and which defaults to off)

Other discussions

@kit-ty-kate brought up an issue with the maintenance branches having a single PR with the changes. The problem is that it's very difficult to suggest to people to test a maintenance release. The sandbox change in 2.0.9 for example, was difficult for users to check (the PR ends up hidden in the list as well). Could we instead have a meta-issue which gets updated with the details, but the change is either pushed directly or done by individual PR.

@rjbou concerned not to lose a central place for the information. @dra27 - we could have it the commit messages when cherry-pick to 2.1 branch reference the meta issue, which would cause GitHub to refer to them. So git cherry-pick -m1 -x.

5 Aug 2021

  • Notes not taken 😲

30 Jul 2021

  • Notes not taken 😲

23 Jul 2021

Agenda

Remaining issues to look at for 2.2.0 planning (from last week):

  • #3737 - is this still an issue?
  • #3722 - uninstalling packages too early when a dependency fails
  • #3709 - ctrl+c in download phase is unreliable
  • #3586 - another system compilers one
  • #3577 - signed sources

16 Jul 2021

Agenda

2.1.0 / 2.0.9 status

Notes from meeting:

  • @rjbou will check through #4742 (opam-version optional field for .changes in 2.0.x) on Monday. cf. OpamFormat.opam_version and it's call in OpamFile.ml (for undefined / defined opam-version fields)
  • At this stage, with other fixes to go, #4744 and #4743 (back-porting the fish fix) can definitely go in

Remaining issues to look at for 2.2.0 planning:

  • #4031 - system compilers upgrade behaviour
  • #3997 - OPAMPRECISETRACKING issues. @dra27 thoughts (possibly posted elsewhere, or maybe I just mused):
    • We could write a file in the opam root which is required to have a non-integral of nanoseconds. Reading that file could confirm to opam that nanoseconds haven't been confirmed. Pros: really simple. Cons: bit hacky.
    • Most files are installed now using opam .install files - i.e. we do read them. So we can compute checksums nearly for free. Why don't we always write checksum and timestamp? In this case, if the timestamp doesn't match then we use the checksum. If that does match, and we have a write lock, we update the timestamp in the changes file? Pros: could be turned on by default. Cons: bit fiddlier to implement.
  • #3983 - behaviour of opam pin when the pinned version matches upstream
  • #3962 and #3928 - issues with non-root use of install.sh. Is that worth looking at in more detail?
  • #3876 - is there anything to do here?
  • #3850 - @avsm's revulsion to {build}. Decision needed!
  • #3849 - csh specific issue. Close?
  • #3833 - I think the use-case is effectively a mixed pin, and so not likely to be supported. Close?
  • #3793 - do we really want to do this?
  • #3766 - adopt XDG for .opam

Notes from meeting

  • 4031: @kit-ty-kate will double-check and then close/escalate as appropriate
  • 3997: possibly the second option should be easy; @AltGr to implement (possibly)?
  • 3850: add a new flag which can be used in dune which indicates to opam that {build} should never be used with it (moves an opam- repo-ci lint check back into opam). @dra27 suggested a global setting to disable the use of {build} attributes completely (for power users). @AltGr not comfortable with this setting. @emillon - we should at least improve the documentation for {build} so that it's easier to understand precisely what it's for.
  • 3793: open a spec for this, but have the links to the (closed) issues in order not to lose the issue.
  • 3766: double-check whether the download-cache should be going in the CACHE part of XDG (the repositories can't). @AltGr warning whether blowing away the Git cache might be an issue.

Unassigned issues in 2.2.0 project

  • #4489 - Tasks in comment; docs and minor CLI
  • #4490 - Tasks in comment; triage and a relatively small tweak
  • #4411 - archive-mirrors tweak
  • #4543 - opam list --recurse default change
  • #4555 - opam init --repositories
  • #4647 - ignoring depends, depopts, conflicts, conflict-class etc. for rebuilds
  • #4576 - Document OPAMEDITOR
  • #4598 + #4613 - lint on missing license + SPDX
  • #4629 - opam install --stop-on-error feature
  • #4649 - PATH manipulation bug
  • #4686 - linting on pin
  • #4688 - opam switch - feature
  • #4737 - checksum on pin-depends
  • #4455 - switch creation when cwd erased bug
  • #4446 - opam list --depends-on including package itself
  • #4419 + #4207- stale files on upgrade bug; repro case to verify and fix to do
  • #4389 - macOS sandbox issue
  • #4305 - parsing invalid hashes in opam file
  • #4277 - testing http backend in CI
  • #4272 - Documentation of the version comparison operator; @AltGr?
  • #4239 - Documentation of exactly how opam determines versions; @AltGr?
  • #4201 - Misconfigured shell scripts -
  • #4749 - sandbox update for CCACHE_DIR. @rjbou?
  • #4528 - new warning when selecting avoid-version packages for the first time

Features without tracking issues (yet)

  • subpath pinning
    • Spec - @rjbou to revise
    • Implementer: @rjbou
    • Reviewer: ??
  • opam file versioning
    • Spec written by @dra27
    • Semi-open question as to how opam determines where to get the plugin from
    • Implementer: ??
    • Reviewer: ??
  • Windows compiler support in opam-repository
    • ocaml-base-compiler upgraded to cope with the two native Windows ports
    • Implementer: @dra27
    • Reviewer: ??
  • Windows depext support
    • Add support for driving Cygwin setup to depext
    • Implementer: @dra27
    • Reviewer: ??
  • Windows shell integration
    • Full support for opam init from the Command Processor and PowerShell
    • Implementer: @dra27
    • Reviewer: ??
  • Semantic Versioning operator
    • Spec required: @AltGr (from issue?)
    • Implementer: ??
    • Reviewer: ??

8 Jul 2021

Agenda

2 Jul 2021

Agenda

25 Jun 2021

Agenda

  • 2.1.0 / 2.0.9 status 🥳
  • Version policy on branches
  • External contributions
  • Issues
    • Tracker items tagged Bug
    • Issues opened this year:
  • Goals

Notes

  • 2.1.0: @rjbou and @AltGr will be on holiday at various points until during w/b 12 Jul. Assuming nothing explodes, we'll tag 2.1.0 next week before @rjbou is off with binaries and release announcement following during w/b 12 Jul.
  • 2.0.9: @dra27 forgot to ask!
  • Version policy for branches. The PR containing commits for releases has been quite useful. The only problem is that not all of us can push to OCamlPro/. Plan: 2.2.0~alpha~dev version bump on master (as done) but no direct pushes or intermediate merges to maintenance branches (apart from unrelated things such as CI fixes). The first person who wishes to cherry-pick to an old branch prepares a PR with the version bump and the cherry-pick and opens the PR; all subsequent cherry-picks are pushed to that PR until release (i.e. the maintenance branch is a series of merges).
  • External contributions/issues: PRs from external contributors all pinged/closed as appropriate. Issues down to #4528 above triaged/reviewed.

18 Jun 2021

The emergency of #4713 stalled announcing the RC this week. The fix in #4715 is passing CI - @rjbou just checking a few more things with registering local switches (e.g. via --switch) but once that's in we can tag rc2 and, in a nod to Groundhog Day, repeat!

Brief discussion on mechanising the binary build process - @AltGr has done something similar with GitHub Actions for Learn OCaml, and is looking at using that for the bulk of the builds for opam as well. Some of the platforms may be trickier (OpenBSD) and either need doing manually, or possibly with custom runners in GHA?

10 Jun 2021

Agenda

  • 2.1.0: confirm release schedule; double-check release criteria
  • 2.0.9: remaining plans
  • LTS, minimum OCaml version and release speed
  • Project Goals
  • 2.2.0: collecting specs
  • New issue: #4690 - tracking issue for Windows-specific problem.
  • New issue: #4688 - opam switch - as restoring the previous switch.
  • New issue: #4686 - when should opam lint warnings be being displayed to users.
  • New issue: #4705 - spaces in parent directory

Notes

2.1.0

  • @dra27 looking at a couple of minor Windows nits - PRs to come.
  • #4705 is serious (and has been present since OPAM 1.1.0!): @rjbou looking at it; @dra27 will sync up with Windows side of it
  • Small display issue with opam lint --help and the lint codes which we happened to spot discussing something else!

2.0.9

Minimum version / LTS / Release cadence

  • opam-core and opam-format to 4.03
  • Rest to 4.08
  • opam-file-format can stay at 4.02.3, but it's on final warning... as soon as there's an issue, it gets bumped

LTS ideas

  • @rjbou: maintaining last release and a nominated LTS release
  • @rjbou: Bug fixes then cause a decision - is master ready to be spun into a release; or do we back-port to last
  • LTS: gets critical bug fixes only
  • Previous release: all bug fixes
  • Nominate at 2.1 as being an LTS, with a view to ensure a release prior to the next Debian freeze (in 2024ish)

Goals

  • Aim to come up with some individual ideas and then go from there
  • If there's good agreement, we may even publish them!
  • Muriel had suggested last year having a poll of users
    • The idea was discussed, but no questions determined
    • We could piggy-back on the OCaml Survey for 2021 - sync with Gabriel Scherer to see if there is going to be one

2.2.0

New issues

  • https://github.com/ocaml/opam/issues/4686
  • Should we be linting when pinning?
    • @altgr - original intention was that the developer would be pinning; perhaps with CI it's less necessary
    • @rjbou - could have an environment variable to list the lint warnings which are disabled (dune-release and opam-publish would need to explicitly disable this)
    • @dra27 - wondering if we could make it that just pin-depends and --dev-repo don't lint?
    • @kit-ty-kate - could display just the errors, not the warnings at pinning time?
    • @altgr - could consider just displaying the errors on opam update (@rjbou - but a bit surprising if you pin in order to test everything's OK!) @dra27 - or could have the linting for local repos only?

21 May 2021

Agenda

Misc

  • #4672 - switches to the dev profile in Dune for CI. Only questions are whether we use the release profile for the builds which run the testsuite and whether everyone's happy to go with it!

The nearly there

The still-to-decide

  • #4668 - 2.0.9+, 2.1.0+ or 2.2.0+?
  • #4673 - confirm that the original command was wrong (i.e. should it have worked)
  • #4662 - original issue addressed by #4671. MacPorts issue described at #4662 (comment)

On the subject of depexts and interactive prompts

  • #4582 - PR itself is just about ready to go, modulo a couple of help messages and a function name
  • Are we strictly implementing the spec properly?
    • In particular, --yes runs sudo commands (as opam depext --yes used to) but is this contrary to Spec for opam depext integration?
    • Should --yes and--confirm-level <> ask imply non-interactive? i.e. do they answer no to the unsafe questions?
    • (this was already an issue with depext and with --unsafe-yes, so it doesn't block #4582 itself)
  • #4669 - does this count as a regression?
  • #4670 - does this count as a regression?

New issues

  • #4674 - proposal to allow packages to request sandbox disabling

Notes

  • 4672: yes, and with running opam-rt and other tests with a release profile build of opam
  • 4663: is ready to merge
  • 4667: @rjbou to review
  • opam-file-format#45: good to go and then release 2.1.3
  • dune-release#343 - @emillon will have a look at the mdx problem; other than that just needs dune build @fmt
  • 4668: OK for 2.0.9
  • 4673: reopened; it is inconsistent that detection works with opam repo add but not with opam switch create; however not for 2.1
  • 4662: original issue is that the depext on "gcc" was incorrect for Macports. Apparently you truly do have to specify the version now. --assume-depexts was the wrong switch to advise. Clarified that --assume-depexts still requires the depext to be listed it just doesn't go ahead and install it. For example, this you allows you to build and make available your own libgmp, but the depext on gmp must still be correct in the opam file.
  • 4671: worth advising there might be a problem with the package itself (i.e. that it's requesting the wrong thing) in addition to suggesting --no-depexts

depext prompting and other interaction

@AltGr and @dra27 in complete agreement: sudo should not be run without a prompt! It's possible that there may be prompts (e.g. sudo prompting for password, etc.). Shouldn't apt be running explicitly with -i if unsafe-yes is not enabled? Conclusion was no: the package manager will prompt if necessary, opam should pause before invoking it and that's enough.

@AltGr that there's an additional confirmation level which is triggered by stdin being closed - it's then no to anything which wasn't being auto-answered. This seems to resolve the question of what should happen for --yes with unsafe questions. For now, they should be interactive, and you can explicitly close stdin (as you can in 2.0) to answer no to those questions. Interaction and confirmation level are not related features - it just happens to be that --no implies there will be no interaction. In 2.2 we might add a flag --non-interactive to allow triggering this behaviour without closing stdin. The suggestion was made that --yes --no actually means "answer yes to safe questions and no to everything else", but even the Windows user vetoed this!

Alpine and EPEL

Alpine issue - let's fix it (@kit-ty-kate looking into it); the alternate command is fast and stops what otherwise looks like a regression (since it works with opam depext).

EPEL. We should view this as a regression simply because there was a hack in opam-depext 2.0 to deal with this. However, there's a simpler and cleaner solution for 2.1: we refine the error message so that if epel-release is mentioned and not installed then we can give me a more precise error message that the user needs to enable the EPEL repo by running dnf install epel-release and then running the opam command again.

This at least limits the special-casing. Users on CentOS 8 already need to have enabled the PowerTools repo manually (even it was done at installation time or by upgrading), and we can both document clearly that EPEL may be required for some depexts and also the error message guides the user to the correct action.

14 May 2021

  • opam-root-version change is nearly ready
  • #4660 opened to track the issue of 2.1 CLI environment variables leaking into package builds (@dra27 will have a look)
  • @kit-ty-kate will rebase and update ocamllabs/dune-release#343 which should mean that both publishing tools are 2.1-ready

7 May 2021

dra27 was a little lax on the note-taking - discussion of active PRs for the 2.1 release candidate

30 Apr 2021

dra27 was a little lax on the note-taking - discussion of active PRs for the 2.1 release candidate

23 Apr 2021

Agenda + notes

Remaining 2.1~rc issues

  • #4640 😱
    • @AltGr live-coded in the meeting...
  • #4638 (closes #4636)
    • See also opam-root-compatibilities
    • Agreeing on the which fields and where is the larger part of this - implementing it should be straightforward!
  • --confirm-level (#4582)
  • Release opam-file-format and update opam to use it (#4639 closes #4394)
    • Two issues being worked by @dra27 (PR by end Friday):
      • It should be possible for opam 2.4, introducing new fields but no new syntax, to use opam-file-format 2.3 for its parsing (i.e. have no 2.4 release), but at present the future best-effort parser would mean that opam 2.4 wouldn't be able to tell the difference between a syntactically valid opam-version: "2.4" file and one which had a parsing/lexing error part-way through. Solved by inserting a sentinel group which can be detected as a syntax error (and has the right location information)
      • The present implementation doesn't return the opam-version line if there's a lexing/parsing error very close (grammatically-speaking) to the version - i.e. opam-version: "2.4" <.
  • #4641 (closes #4619)
  • #4642 (addresses #4554
    • Include in 2.1? Doesn't address the issue with opam-state 2.0.x and opam binary 2.0.x, but it would mean that plugins and tools using opam-state 2.0.x and opam binary 2.1.x won't see the same issue
    • Agreed - low-risk internal change, which will stop the appearance of a performance regression (if opam-state 2.0.x starts fighting with opam 2.1 over the cache)
  • ocaml-mccs release done - need to update lib-ext
  • #4643 - OK to include, but make cold shouldn't include the solver, just as we're quite close to rc. No problem to add an additional cold target (e.g. cold-0install) then it can be added via that.

Release (candidate) steps

  • dune-release is not yet compatible with opam-state 2.1 - ocamllabs/dune-release#343
  • opam-publish is not yet compatible with opam-state 2.1 - no PR?

16 Apr 2021

Agenda

Remaining 2.1~rc issues

  • --confirm-level (#4582) or --unsafe-depext-yes (#4563)
  • #4636 needs implementing and back-porting for 2.0.9
  • Release opam-file-format and update opam to use it (closes #4394)
  • Implement plugin symlink erasing at upgrade (closes #4619)
  • Fast install (#4494) just needs the the conflict resolved, I think?
  • @AltGr/@kit-ty-kate time to rebase (closes #4545)
  • An ocaml-mccs release is advisable to pick up #30 (4.12 support)

Release (candidate) steps

Other news

  • @AltGr - some work with the Software Heritage Foundation has been agreed (with funding for OCamlPro) archiving all packages from opam-repository there (not just the GitHub repos). Also looking at support for referencing SHF copies in the opam files themselves (i.e. with an ID) - could include falling back to their caches.

1 Apr 2021

Agenda

New PRs

#4619 - plugins and opam-state and upgrades

  • Plugins symlink reset at upgrade (together with #4621 this fixes the issue)
  • Tweaking opam-state to allow read-only access to new opam roots, when possible. Suggested in #4266 but dropped by the implementation in #4280 (following discussion at 17 Jul 2020 dev meeting, based on time stamps of messages!). As we encourage more plugins to incubate ideas, I think we should revisit this: idea is that the config files in the root (global, switch and repo) all have opam-root-version: "2.1" and opam-version: "2.0". The idea is simple: opam-root-version is checked for taking out write locks, opam-version is checked for taking out read locks and opam-state 2.0.x should be updated to internally recognise opam-root-version (i.e. no change in API in OpamFile) and ignore fields it doesn't recognise. The changes we've made for 2.1 mean that it should "just work" to have readable roots to 2.0.

26 Mar 2021

Agenda

Notes in italics

Issues & PRs

No new issues! \o/

2.1.0

  • opam-file-format#43
    • @dra27 to get 4.02 working, but otherwise ready for review
  • opam-file-format#32 - needed/safe for 2.1 release?
    • @rjbou - not urgent for opam 2.1; can double-check with @NathanReb whether it's still urgent for dune-release. Primary problem is that it keeps the "generated by dune comment". Given that it's possible to do it from the lexer, should this code go to dune-release and potentially be spun out further? Further discussion - potentially moving OpamPrinter.Preserved to a separate library.
  • #4616 (not yet opened!) (@dra27) - cold compiler to 4.12.0 to fix Cygwin (and some src_ext updates)
  • #4612 - replaces #4599
    • @AltGr to review
  • #4607 (@AltGr)
  • #4606 (@rjbou)
  • #4591 (@AltGr)
  • #4582 (@rjbou) - alternative to #4563
    • blocked on various CLI versioning PRs
  • #4581 (@rjbou) - blocked on #4606
  • #4580 (@rjbou)
  • #4563 (@rjbou)
  • #4548 (@kit-ty-kate)
    • move to next
  • #4494 (@kit-ty-kate)
  • #4438 (@rjbou)
  • #3897 (@dra27)

2.0

  • #4547 (@rjbou) - 2.0.9 release
  • #4538 (@kit-ty-kate) - CUDF trimming for 2.0

19 Mar 2021

Agenda

PR Sweep

Notes in italics

macOS testing CI problem with opam-rt - a sandboxing script update seems to have broken macOS testing?

2.1.0

2.0

External contributors

Misc. / next

12 Mar 2021

Agenda

Notes

  • Folding the ocaml-option-* packages into ocaml-base-compiler. @kit-ty-kate - concerned potentially at ocaml-option-vanilla not being default. Let's discuss further on a PR.

  • Using 4.13.trunk - @AltGr should be double-checked just in case the letters are treated separately. @kit-ty-kate - the version numbers do work.

  • For ocaml-src.4.13+trunk, @kit-ty-kate - what would happen if you wanted to use the dev branch after 4.13.0 is released? @dra27 - surely that wouldn't be needed for this package, though; the package is there so that there's one for the release.

  • OK to merge the 2020 Dev Meetings into a single page? The only downside would be the loss of linking. @rjbou - use a 2021 page straight away; @dra27 - with anchors for the dates.

26 Feb 2021

Agenda

19 Feb 2021

Present: @avsm (notes), @kit-ty-kate, @rjbou

  • all focus on rc next week -- looking at blockers.

  • start pushing out stuff to 2.1.1 milestone as we need to cut the RC.

  • https://github.com/ocaml/opam/pull/4542 is going to go in shortly.

  • https://github.com/ocaml/opam/pull/4311 is a bad performance regression, but we could punt this to 2.1.1 if we dont get a fix in time.

  • https://github.com/ocaml/opam/pull/4514 needs a decision on which variables we should take. Pushing out to 2.1.1

  • https://github.com/ocaml/opam/issues/4321

    • cant design a confirm-level feature in time for 2.1.0 - defer to 2.2.0 and a design spec
    • rename the cli option to explicitly note that it is dangerous, so it doesnt show up in tutorials etc: opam install --yes --unsafe-depext-yes

  • Minimum OCaml version for 2.x

    • bump to 4.06 as the minimum version for the 2.x series
    • we have bootstrap from a C compiler for any distros that are < 4.06 and not upgrading.

12 Feb 2021

Notes

@dra27 - Confirming that semver changes will wait until post 2.1, please - just for safety! @samoht has a point that we should discuss whether it's what's wanted

@emillon mentioned that Dune 3 development is starting. Since it's major, there are features slated for removal, including dropping support for OPAM 1.x - mainly finding the switch. @dra27 - it sounds fine (@AltGr agrees too) - maybe use opam --version in the error message to make it friendly that it's OPAM 1.2.2

@emillon - what should {with-doc} should do. @AltGr it's simply a variable, from opam's perspective. At present, many projects do @doc {with-doc} but this builds documentation without installing (@dra27 - which is of course wasted effort!).

@kit-ty-kate - does opam know when a package was installed using with-doc? @AltGr - no, it's not recorded @dra27 - although it's likely to change with package variables

@kit-ty-kate - discussions with @samoht on the difference in semantics between opam list --with-test and opam install --with-test. @dra27 - needed for 2.1? @kit-ty-kate - no, too big. @dra27 - let's ensure it's tracked to be picked up later. @kit-ty-kate - opened #4543

  • Various issue and PR triages - directly commented on the issues themselves

29 Jan 2021

Notes

Kate

  • Welcome to maintainership! @avsm will sort out GitHub permissions later.

Minimum supported version?

  • Dose6 is only just out - let's stick both 2.0 and 2.1 with Dose5, but switching to ocamlgraph 2 should be done (@AltGr agrees - it would be worth double-checking changes)
  • We may need to carry a few patches for ocamlgraph 2 in lib-ext
  • The release of extlib 1.7.7-1 unblocks 2.0.8 for OCaml 4.12, so that should be good to go

New issues

  • All in hand!

2.1 roadmap to RC

  • Ticking forward
  • @emillon note for 2.2 for removing unused fields from the format (Slack discussion this week - will move to an issue in the 2.2 project so we track it).

22 Jan 2021

Agenda

Notes

opam 2.0.8

  • Tests all working - backport PR is ready to merge
  • Should be working with OCaml 4.12~beta1 - @rjbou to check before release

New issues

  • #4505 - @rjbou to determine whether to fix for 2.1
  • #4507 - @rjbou problem would be calling the solver each time to check the dependencies, so the call would be much slower. @AltGr - another possibility would only to check the direct dependencies with the new flag. @dra27 - a third possibility would be to note the predicate values and see that they've changed and so interpret that as a reinstall. This is related to package parameters and so might be "fixed in passing". @dra27 to update issue
  • #4503 - restore opam config var for the RC; but ensure PRs open to use opam var for the identified tools

opam 2.1 rc checks

opam-file-format

  • @dra27 to have a look at #4394 (ensuring that opam-version: "2.1" and greater is the first non-whitespace/comment line)
  • @rjbou will ensure ocaml/opam-file-format#32 is merged for 2.1.3
  • ocaml/opam-file-format#25 is non-critical

mccs

  • AltGr/ocaml-mccs#31 may be fixed by a newer GLPK
  • Should check that GLPK 5.0 works, since Debian will package it at some point

src_ext

  • Dose 6 is nearly released - @AltGr
  • ocaml/opam#4470 adds required support for ocamlgraph 2.0

opam-lib users

  • opam-publish is updated
  • dune-release - @kit-ty-kate has a branch (https://github.com/kit-ty-kate/dune-release/tree/opam-2.1) but not yet a PR which has been being tested
  • Camelus - doesn't appear to be updated (@ThomasBlanc is away for a while) - but this is perhaps less critical. @AltGr to have a quick look and check the status?

CI using opam 2.1

  • ocaml-ci is using 2.0.7 - but it's based on docker-base-images which does have a 2.1 binary in it. Kate will test.
  • opam-repo-ci is using 2.1 with no problems
  • ocaml-ci-scripts - @dra27 will do a PR to set OPAMCLI=2.0
  • setup-ocaml - @dra27 will check it's uses

opam-repository-mingw

  • @dra27 + @MisterDA will look into that

opam upgrade

@emillon - it's presently difficult for both developers to test out the betas. We should have some more visible documentation on the process and potentially a tool to help. @AltGr - the install.sh script is supposed to do backups and other things. @AltGr - see "Try it!" at end of https://opam.ocaml.org/blog/opam-2-1-0-alpha/ @rjbou - there is also https://github.com/ocaml/opam/wiki/How-to-test-an-opam-feature (but it's a bit hidden away).

@dra27 - the wiki suggests having an alias for the new binary, but perhaps it would be better to have one for the old binary. So you have a backup of the old opam root and also a way to get back to the old root if necessary.

@emillon - what about local switches, as they're outside the root. @AltGr - the local switches are listed in the root (and this survives the upgrade). @emillon - what about if the local switch is used with 2.1, can it then be used with 2.0? @AltGr - almost certainly not, because the format may be upgraded.

@dra27 - rather than backing up local switches, perhaps we should just warn that that change is one way, so you might have to remake the local switch if you go back

@emillon - yes, the main point is to be clear exactly what backing up means and we do and don't get.

15 Jan 2021

Agenda

Notes

Present: @AltGr, @dra27, @kit-ty-kate, @emillon

beta4 -> rc

@AltGr: various things moving around with the conflict messages which are nearly there.

  • #4344 - @AltGr reckons this is closable.
  • #4489 - docs update; can be pushed to 2.2 if necessary
  • #4381 - @AltGr - almost certain a 1.2.2 upgrade hangover (initialise the defaults) so could be reclassified as a bug - you should just end up with empty selections. @dra27 will update the issue.
  • System compiler upgrading - this should be working; there should be a workflow for it to test it @dra27 will open and issue
  • Setting up switches for "OCaml 4.12" or "OCaml 4.11" with automatic upgrade to point releases. @AltGr - default at the moment will not do this. It's not obvious what the change would look like - let's add this to the considerations for opam 2.2.
  • #4188 - @rjbou to check and update; @AltGr - this may well have been fixed already (the problem was the code to infer the new switch invariant triggering in a read-only context)
  • #4172 - issue is still present; performance improvement bump to 2.1.1 @emillon - may be able to improve the performance of this by changing opam show slightly. For example, local switches are often unnecessarily scanned.
  • #4311 - slowdown appears to be caused by the switch invariants (@AltGr so it's slower "by design"!). @dra27 - can something be done either with two solves; one to get and fix the invariant and then with that list of packages in the base and then a second solve. @AltGr - for --installable the external solver isn't involved; this is actually Dose's SAT solver. @kit-ty-kate - the switch invariant in these cases is ocaml-base-compiler (it's come via an upgrade). @kit-ty-kate - --co-installable-with ocaml-base-compiler.4.11.1 speeds things up considerably. @AltGr - the universe is encoded differently which affects this, too (although it's not clear why that would be responsible for this slow down). @dra27 - does this slowdown still occur even when the switch invariant is exactly the list of base packages from the 2.0 switch? @kit-ty-kate - nope! @AltGr - it sounds as though --co-installable-with is usually what the user wants. @dra27 - in fact, given that --no-switch was needed in 2.0, would it be better for --installable in 2.1 to be assuming the base packages.
  • #4305 - @dra27 will update issue - definitely to be fixed; but not 2.1
  • #4313 - @AltGr will look into it - the two sequences of commands should be equivalent.
  • #4201 - remove from the 2.1 milestone
  • #4245 - remove from the 2.1 milestone
  • #4333 - link to the new issue @kit-ty-kate added
  • #4400 - @AltGr testing
  • #4389 - not critical for 2.1
  • #4454 - @rjbou to determine
  • #4455 - not critical for 2.1
  • #4485 - remove from the 2.1 milestone
  • #4490 - remove from the 2.1 milestone
  • #4297 - @rjbou to determine

8 Jan 2021

Agenda

  • opam-monorepo demo and discussion for metadata with @NathanReb
  • opam-file-format 2.1.2 & opam 2.1 beta4
  • Switching off Travis - is everything migrated?
Clone this wiki locally