Proposal for Dune Plugins

[jonahbeckford]

Dune Plugins

• Dune Plugins
  • Problem
  • Terms
  • High Level Requirements
  • Implicit Proposal through Examples
    • Capnp Code Generation
    • vcpkg C Library Dependency Injection
  • Formal Proposal
  • Artifacts

Problem

Given there is a limited set of core Dune developers, it may take years for new tooling to be integrated into Dune.

Terms

• sexp: https://github.com/janestreet/sexplib#sexplib---s-expressions-for-ocaml. Example: (lang dune 3.8)
• whitespace concatenated sexp: zero or more sexp concatenated with whitespace. Example: (foo.bar) (baz.wonky)

High Level Requirements

1. H1 - Do not violate Dune's composability tenets. In particular, there should only no coupling, or at most a loose coupling, between plugins. (@rgrinberg)
2. H2- The plugins should be usable independently of Dune. (@xavierleroy - https://discuss.ocaml.org/t/a-roadmap-for-the-ocaml-platform-seeking-your-feedback/12238/55?u=jbeckford)

Implicit Proposal through Examples

Capnp Code Generation

More Terms:

• Capn' Proto: https://github.com/capnproto/capnp-ocaml#capnp-ocaml

What? Tooling to generate .ml files from Capn' Proto .capnp schemas.

Generalizes to? Any tool that generates .ml, .mli, .mll, .mly and/or .mld from
another type of file.

Without this proposal, a Dune user would add a Capn' Proto generated .ml and .mli files by using the following Dune stanzas (confer: https://github.com/capnproto/capnp-ocaml/tree/v3.6.0/src/examples/1) in her dune file:

(executable
  (name example1)
  (libraries capnp))

; This rule tells dune how to generate foo.ml and foo.mli by compiling foo.capnp against the CAP'N PROTO compiler
; PLEASE NOTE that capnpc must exist on your computer for this to work (https://capnproto.org/install.html)
(rule
  (targets foo.ml foo.mli)
  (action (run capnp compile -o %{bin:capnpc-ocaml} %{dep:foo.capnp})))

and in her dune-project file:

(lang dune 4.0)
(package
  (name example)
  (synopsis "An example")
  (depends
    (capnp (>= 3.5.0))))

and manually compile and install the C++ executable capnp using instructions on https://github.com/capnproto/capnproto.

With this proposal, a Dune user would instead write in her dune file

(executable
  (name example1)
  (libraries capnp))

(capnp.generate (name foo))

and in her dune-project file:

(lang dune 4.0)
(using capnp 0.1)
(package
  (name example)
  (synopsis "An example")
  (depends
    (capnp (>= 3.5.0))
    ; Question: Dune should be able to infer the opam package name directly from its use
    ; in (capnp.generate). Does the user have to explicitly add "dune-capnp" at all?
    (dune-capnp (>= 0.1.0))))

The capnp.generate plugin author would be responsible for:

1. Creating an opam package dune-capnp.0.1.0 which installs the executable dune-capnp.exe into the opam switch and queryable from findlib. An ordinary dune-project based package can trivially do this, but nothing precludes non-Dune based packages that conform to findlib.
2. Dune will invoke dune-capnp.exe translate-sexp EACH TIME Dune encounters (capnp.generate ...). The functional specification is:

   1. The standard input will be the contents of the Dune stanza:

      (source
        (project_dir (where /proj1))
        (project_pwd 
          (where /proj1/src)
          (files dune foo.capnp foo.ml foo.mli)
          (dirs d e f)))
      (context
        (name default.linux_arm64)
        (project_root /proj1/_build/default.linux_arm64)
        (pwd /proj1/_build/default.linux_arm64/src))
      (input
        (loc
          (start (pos_fname "dune") (pos_lnum 123) (pos_bol 7) (pos_cnum 8))
          (end (pos_lnum 123) (pos_bol 67) (pos_cnum 68)))
        (text "(capnp.generate (name foo))"))

      Dune has a set of exclusions that must be respected by the plugin; the
      (source (project_pwd (files ...) (dirs ...))) captures that. In addition
      the plugin only has read-only access to source files in the current directory (project_pwd).

      The (context) is the currently active build context. Any files in <project_root>/plugins/capnp generated by the dune-capnp.exe initial-sexp command (which will be documented further below) will be available to dune-capnp.exe translate-sexp.

      The (input) contains location information for error reporting, and the raw text of the sexp to be translated. It is raw text so that whitespace is preserved for error reporting.

   2. The arguments will be:

      1. argv[0] = dune-capnp.exe
      2. argv[1] = translate-sexp
      3. argv[2] = generate. This is from (capnp.generate ...).
      4. argv[3] = --plugin-version=0.1. This is from (using capnp 0.1).
      5. argv[4] = --dune-lang=4.0. This is from (lang dune 4.0).
      6. argv[5] = --protocol=1. See the Formal Proposal.

   3. The standard output (the response of dune-capnp.exe) will be of the form:

      (translate
        (error
          (message "If error it goes here")
          (loc
            (start (pos_lnum 123) (pos_bol 7) (pos_cnum 8))
            (end (pos_lnum 123) (pos_bol 67) (pos_cnum 68))))
        (output
          (rule
            (targets foo.ml foo.mli)
            (action (run capnp compile -o %{bin:capnpc-ocaml} %{dep:foo.capnp})))))

      The response will either have an (error) or it will have an (output), but not both.

Because the plugin needs a C executable capnp in the PATH only during the build, the capnp.generate plugin author will also be responsible for:

1. Dune will invoke dune-capnp.exe initial-sexp ONCE, regardless of the number of occurrences of (capnp.generate ...). The functional specification is:

   1. The standard input will be the contents of the Dune stanza:

      (project
        (project_dir /proj1))
      (contexts
        ((name default)
          (project_root /proj1/_build/default)
          (ocaml_config (host x86_64-pc-linux-gnu) (target x86_64-pc-linux-gnu)))
        ((name default.linux_arm64)
          (project_root /proj1/_build/default.linux_arm64)
          (ocaml_config (host x86_64-pc-linux-gnu) (target aarch64-pc-linux-gnu))))
      (input
        (loc
          (start (pos_fname "/proj/1/dune-project") (pos_lnum 123) (pos_bol 7) (pos_cnum 8))
          (end (pos_lnum 123) (pos_bol 67) (pos_cnum 68)))
        (text "(using capnp 0.1)"))

      The (contexts) contain the one or more build contexts. Each field in a build context comes from the publicly declared Dune Variables; they are not all listed here for brevity's sake.

   2. The arguments will be:

      1. argv[0] = dune-capnp.exe
      2. argv[1] = initial-sexp
      3. argv[3] = --plugin-version=0.1. This is from (using capnp 0.1).
      4. argv[2] = --dune-lang=4.0. This is from (lang dune 4.0).
      5. argv[3] = --protocol=1. See the Formal Proposal.

   3. The standard output (the response of dune-capnp.exe) will take the form:

      (initial
        (error
          (message "If error it goes here")
          (loc
            (start (pos_lnum 123) (pos_bol 7) (pos_cnum 8))
            (end (pos_lnum 123) (pos_bol 67) (pos_cnum 68))))
        (output
          ; TODO: This is pseudo-code for building the C++ capnp executable, using
          ; TODO: direct translation of the standard capnp instructions. Obviously CMake
          ; TODO: won't work, but lucky that capnp supports autoconf and Bazel as well.
          ; TODO: So it should be possible to translate the build instructions to
          ; TODO: a Makefile or a script. The real code is not germane to this discussion.
          (rule
            (targets capnp%{ext_exe})
            (action (progn
              (run cmake
                  -S %{lib:dune-capnp:vendor/github.com/capnproto/capnproto}
                  -B build/
                  -D "CMAKE_C_COMPILER=%{ocaml-config:c_compiler}"
                  -D "CMAKE_CXX_COMPILER=%{ocaml-config:c_compiler}"
                  -D "BUILD_TESTING=OFF"
                  -D "CAPNP_LITE=ON")
              (run cmake --build build/)
              (run cmake --install build/ --prefix installdir/)
              (run copy installdir/bin/capnp%{ext_exe} %{target}))))))

      The response will either have an (error) or it will have an (output), but not both.

vcpkg C Library Dependency Injection

More Terms:

• vcpkg: A popular C package manager; there are others. https://vcpkg.io/

• Dune Configurator: A Dune library that lets a Dune project (typically)

  1. Discover C flags, linker flags, etc.
  2. Capture those flags into one or more sexp files

  Those sexp files can then be used in Dune clauses throughout the Dune project.
  https://dune.readthedocs.io/en/stable/dune-libs.html#configurator

• dependency injection: A pattern where dependencies are given to an entity (ex. a Dune project). The pattern constrasts with an entity instantiating or discovering its own dependencies (ex. Dune Configurator).

What? Tooling to inject C flags and linker flags for a C library from a Dune workspace into Dune projects, especially into child (vendored) Dune projects.

Generalizes to? The obvious narrow extension is to any C dependency injector tool. But more generally any tool that needs access to the cross-compiling context or source code files.

Without this proposal,

1. The maintainer of the OCaml sqlite package (an arbitrary example) would write code that uses the Dune Configurator to discover where the C library libsqlite3 was located. The maintainer would be able to find sqlite3 through pkg-config on Unix systems if it were pre-installed.

   (* file: discover.ml *)
   module C = Configurator.V1
   let () = C.main ~name:"sqlite3" (fun c ->
      let cflags, clibs = match C.Pkg_config.get c with
      | None -> failwith "pkg-config not found"
      | Some pc ->
         (C.Pkg_config.query pc ~package:"sqlite3" with
         | None -> failwith "sqlite3.pc not found"
         | Some conf -> conf.cflags, conf.libs)
      in
      C.Flags.write_sexp "cflags.sexp" cflags;
      C.Flags.write_sexp "clibs.sexp" clibs);

   ; file: dune
   (executable
     (name discover)
     (libraries dune-configurator))
   (rule
     (targets sqlite3.loc cflags.sexp clibs.sexp)
     (action (run ./discover.exe)))
   (library
     (name sqlite3)
     (foreign_stubs (language c) (names caml_sqlite3_stubs.c)
       (flags (:include (cflags.sexp))))
     (c_library_flags (:include (clibs.sexp))))

2. The Dune user of the sqlite3 package would write ...

   (* file: discover.ml *)
   module C = Configurator.V1
   let () = C.main ~name:"my-batch-processor" (fun c ->
      let exe = match C.which c "sqlite3" with
      | None -> failwith "sqlite3 binary not installed"
      | Some exe -> exe
      in
      C.Flags.write_lines "sqlite3.loc" [exe]);

   ; file: dune
   (executable
     (name discover)
     (libraries dune-configurator))
   (rule
     (targets sqlite3.loc)
     (action (run ./discover.exe)))
   (executable
     (name my-batch-processor)
     (libraries sqlite3))
   (cram (deps
     %{read-lines:sqlite3.loc} ; make sqlite3 binary available in cram tests
     create-tables-and-do-inserts.sql
     my-batch-processor.exe))

   # file: test.t
   Create database + populate entries
     $ sqlite3 -bail test.db ".read create-tables-and-do-inserts.sql"
     $ sqlite3 test.db "SELECT COUNT(*) FROM queued_images;"
     7
     $ sqlite3 test.db "SELECT COUNT(*) FROM completed_images;"
     0
   
   Exercise OCaml code
     $ my-batch-processor.exe -db test.db
     $ sqlite3 test.db "SELECT COUNT(*) FROM completed_images;"
     7

Everything in the following list increases the difficulty:

• use opam's depext to install sqlite3 if it were not present
• download the sqlite3 source code if they needed a specific version (the luv package does this)
• get sqlite3 working on Windows and other systems that don't have a system package manager
• get sqlite3 working with dune build -x cross-compilation

With this proposal,

1. The maintainer of the sqlite3 would write:

   ; file: dune-project
   (lang dune 4.0)
   (using c_inject 0.1)
   
   ; file: dune
   (c_inject.try_vcpkg         (name sqlite3))
   (c_inject.try_pkgconfig     (name sqlite3))
   (c_inject.fail_if_not_found (name sqlite3))

   to make:

   • lib/libsqlite3.a and lib/libsqlite3.so available if the Dune workspace supports vcpkg
   • The results of pkg-config --libs sqlite3 (ex. -lsqlite3) if the Dune workspace does not support vcpkg

   The maintainer would also write:

   ; file: dune
   (library
    (name sqlite3)
    (foreign_stubs (language c)
      (names caml_sqlite3_stubs.c)
      (flags (:include (%{project_root}/c_inject.sqlite3.cflags.sexp))))
    (foreign_archives (:include (%{project_root}/c_inject.sqlite3.archives.sexp)))
    (c_library_flags (:include (%{project_root}/c_inject.sqlite3.clibs.sexp))))

2. The c_inject plugin author's responsibility would be to create the sexp files
   %{project_root}/c_inject.<name>.{cflags,archives,clibs}.sexp.

3. The Dune user of the sqlite3 package would write ...

   A vcpkg.json file for making the sqlite3 libraries with the specified sqlite3 features enabled available in either Dune's "default" context (if no cross-compiling) or in Dune's cross-compiling context (if there is only one cross-compiling context):

   {
    "name": "my-batch-processor",
    "version": "0.0.1",
    "dependencies": [
      {
        "name": "sqlite3",
        "features": ["geopoly", "math"]
      }
    ]
   }

   Because vcpkg only supports one cross-compiling toolchain, there would be an error if Dune supplied more than one cross-compiling context.

   A vcpkg-host.json file making the sqlite3 executable:

   {
    "name": "my-batch-processor-host-tools",
    "version": "0.0.1",
    "dependencies": [
      {
        "name": "sqlite3",
        "host": true,
        "features": ["tool"]
      }
    ]
   }

   And the other files:

   ; file: dune
   (vendored_dirs sqlite3-src) ; sqlite3-src/ is source code for sqlite3
   (c_inject.try_vcpkg         (name sqlite3)  ; for my-batch-processor
      (executables sqlite3 sqlite3_analyzer sqldiff)
      (host true))
   (c_inject.try_vcpkg         (name sqlite3)) ; for sqlite3-src
   (c_inject.try_pkgconfig     (name sqlite3)) ; for sqlite3-src
   (c_inject.fail_if_not_found (name sqlite3))
   
   (executable
     (name my-batch-processor)
     (libraries sqlite3))
   (cram (deps
     %{project_root}/c_inject.sqlite3.host-executable.sqlite3.txt
     create-tables-and-do-inserts.sql
     my-batch-processor.exe))

   # file: test.t
   Create database + populate entries
     $ sqlite3 -bail test.db ".read create-tables-and-do-inserts.sql"
     $ sqlite3 test.db "SELECT COUNT(*) FROM queued_images;"
     7
     $ sqlite3 test.db "SELECT COUNT(*) FROM completed_images;"
     0
   
   Exercise the cross-compiled OCaml code with emulator pseudo-code
     $ qemu-img create -f raw hda 8G
     $ qemu-system-aarch64 -name "sles" -machine type=pc,accel=kvm -m 768 \
       -smp 2 -boot c \
       -drive file=hda,if=virtio,index=0,media=disk,format=raw \
       -virtfs local,path=.,mount_tag=host0,security_model=passthrough,id=host0 \
       -net nic,model=virtio,macaddr=52:54:00:05:11:11 \
       -device virtio-net-pci,netdev=hostnet0,hostfwd=tcp::2222-:22 \
       -vga cirrus -balloon virtio
     $ ssh localhost -p 2222 mount /dev/host0 /mnt/host0
     $ ssh localhost -p 2222 /mnt/host0/my-batch-processor.exe -db /mnt/host0/test.db
     $ sqlite3 test.db "SELECT COUNT(*) FROM completed_images;"
     7
     $ ssh localhost -p 2222 shutdown

Formal Proposal

• A Dune macro fully replaces the (<macroname> ...) stanza with the expansion of the Dune macro
• Each (name1.name2 ...) Dune stanza in a dune file where the head of the stanza contains a dot (ex. capnp.generate) is a Dune macro call implemented in a Dune plugin executable named dune-<name1>.exe.
• The expansion of a Dune macro is dictated by Dune communicating to a Dune plugin using Protocol 1:
  • The Dune macro call expansion is the standard output of dune-<name1>.exe with arguments translate-sexp <name2> --protocol=1 --dune-lang-VERSION. The Implicit Proposal specifies the translate-sexp sub-command.
  • Before the first Dune macro call of (A.B ...), Dune will call the initial-sexp --protocol=1 --dune-lang-VERSION subcommand and inject its standard output in the dune-project directory scope. The Implicit Proposal specifies the initial-sexp sub-command.
  • Protocol 1 is inefficient because it spawns a new process for every matching Dune stanza, but is simple.
  • The Dune plugin exit code 13 is reserved to communicate that --protocol=N is not supported by the Dune plugin. An optimized Protocol 2 may be written at a future date which uses the Dune RPC protocol. Of course that is overkill in the early stages of authoring a Dune plugin. Dune should try the highest protocol it supports, and if it gets exit code 13 it should downgrade the protocol number until it no longer receives exit code 13.
• The initial-sexp and translate-sexp <name2> output of a Dune plugin <name1>:
  • Is zero or more Dune stanzas that have been serialized as whitespace concatenated sexp
  • Must conform to the implicit or explicit Dune schema (lang dune VERSION) where VERSION is passed in the option --dune-lang=VERSION.
  • May contain Dune stanzas that are Dune plugin invocation. For example, (foo.bar ...) may emit (baz.wonky ...), where foo is one plugin and baz is another plugin. That is, Dune plugins are context-independent in their inputs to satisfy high level requirement H1 but may be context-dependent in their outputs.

Artifacts

• Dune plugin authors need some Dune stanza creation library to emit correct Dune stanzas that conforms to a particular (lang dune VERSION). That library must be usable in non-Dune build systems to satisfy high level requirement H2, and ideally would be owned by the Dune team. I have a library that might be a starting point: https://v3.ocaml.org/p/dkml-dune-dsl/latest/README.md.html.

[rgrinberg]

Thanks for the interesting and well thought out proposal. Some things are not too different from the various discussions about plugins the team had over the years.

I'm going to point a few instances where I think this proposal deviates from dune's normal design patterns or fails to address important (IMHO) use cases.

1. In dune, it's customary to enable extensions with a particular version. For example, (using capnp 1.5). The expectation is that a concrete version of the plugin should support more than one version of its schema. In particular, extensions should version themselves independently from dune. The plugin should also know how to tell the user which versions of its own schema it supports.

2. The dune lang version that is passed from dune to the plugin is logically the version of the dune language the user is using to communicate with dune. I find it a bit inelegant that this is the same version the plugin must use to generate stanzas. It seems like plugins will need to understand quite a bit about dune's stanza language to work with all the various versions of the dune language that are out there in the wild. I don't have a better suggestion here.

3. In general, there's no discussion of error messages. At the very least, the plugin should be able to signal errors with appropriate locations to the user just like dune does.

4. As far as I can tell, a plugin has no way of querying dune for information. For example, a plugin might want dune to answer questions like "what context am I running in?", "what is the source file where I'm generating stanzas?". More complex plugins might even want to do IO by asking dune to "build file X and give me the contents".

5. Dune already has an RPC protocol and I think it would be better if the "persistent request/reply" protocol would be built on top of it.

Finally, you've already pointed this out as a concern, but we need a naming scheme that allows dune to go from a plugin name to the package where this plugin must exist. Without this convention, we won't have effective lazy loading.

[jonahbeckford]

Thanks for the feedback. I will incorporate it hopefully within a week.

[jonahbeckford]

Took a while but I incorporated most of the feedback. I also added a lightly-sketched second example (vcpkg) to tease out what extra information a plugin might need (feedback item 4).

[gridbugs]

Thanks for the proposal! I have a few comments/questions.

Firstly I want to discus the ux of declaring that a project uses a plugin. Consider this example:

(lang dune 4.0)
(using capnp 0.1)
(package
  (name example)
  (synopsis "An example")
  (depends
    (capnp (>= 3.5.0))
    ; Question: Dune should be able to infer the opam package name directly from its use
    ; in (capnp.generate). Does the user have to explicitly add "dune-capnp" at all?
    (dune-capnp (>= 0.1.0))))

The main confusion I have about this as a user is that the plugin is enabled for the entire project but the package implementing the plugin is listed in the dependencies of a specific package. My first question would be what is the behaviour if dune-capnp is omitted from the list of dependencies? Also, what happens if a second package is added to the project but only one package depends on dune-capnp?

As you mentioned, we could have dune automatically add dependencies to packages when it sees that they use a plugin and use a naming scheme like dune-<name> for the package name. I like this idea because then only the packages in a multi-package project that actually use the plugin end up depending on its package.

An alternative might be to add a keyword for running arbitrary executables (installed from dependencies) that implement the plugin protocol. E.g.

(plugin dune-capnp.exe generate (name foo))

Before evaluating rules dune would expand all the plugin stanzas by running the exes (dune-capnp.exe installed from the dune-capnp package). This would remove the need to explicitly register a plugin with dune (the (use capnp 0.1)) and be more transparent in its operation. We could mangle the exe name to remove the "dune-" prefix and the extension to enforce a naming convention on exes that implement dune plugins (kind of like how all cargo extensions are just programs named cargo-<name>) to make the plugin statements more concise. This solution would also remove the need for a scheme for determining the package name from the plugin name and allow multiple plugins to be implemented in the same package, and for the package to be named whatever the author wants.

Regarding the matter of generating dune syntax of the appropriate version, @rgrinberg what do you think about making a library to help with this sort of thing, where users could construct typed ASTs and then convert them into sexps compatible with a given version of lang dune?

[rgrinberg]

Doing it per project rather than per package is perhaps not ideal, but the it would simplify the implementation tremendously. Dune already manages plugins and the version of the dune language at the project rather than the package level. Finally, the question of "what package I'm in" cannot even be answered in certain situations with dune files. While the question "what I'm project I'm in" is always easy to answer. I do acknowledge your point that this approach is somewhat wasteful though.

Regarding the matter of generating dune syntax of the appropriate version, @rgrinberg what do you think about making a library to help with this sort of thing, where users could construct typed ASTs and then convert them into sexps compatible with a given version of lang dune?

Seems like a good idea.

[davesnx]

Haven't received any feedback about (Provide a way to call dune programatically #8638) but it's conceptually fixing the same extensibility problem.

The design of a plugin system is indeed a little tricky, allowing a free-form for users might eventually expose easier what are the limits where users are trying to push dune forward.

Also, both "features" can co-exist.

If it's an idea that it's worth pursuing I might have time to create a proper RFC for this.

[rgrinberg]

I would say there's one main difference between the two proposals:

1. The plugin system is going to have strong backwards compatibility guarantees
2. Using dune as a library will enjoy access to a much larger API, with the caveat is that a lot of it will be unstable.

[rgrinberg]

Thanks for the updated proposal. One point I would disagree with is that using RPC is going to be overkill for the first version. If we're going introduce a library for plugin authors, we might as well make this library handle communicating with dune using the existing RPC mechanism. This is an already working building block we can just use, so it will simplify the implementation of a library we already plan to introduce. Not to mention that it will make it far simpler to evolve things.

The rest of your proposal seems fine to me. I'm still a bit uneasy about the naming scheme to accommodate lazy loading. Another option is to just to make users write the package name from which a plugin must come from in their project file.

In any case, I think this proposal is good enough to start gathering feedback from the rest of the interested parties. I will ask in the dev channel to comment.

[jonahbeckford]

One point I would disagree with is that using RPC is going to be overkill for the first version.

I had a look at the RPC example at https://github.com/ocaml/dune/blob/main/otherlibs/dune-rpc-lwt/examples/rpc_client/rpc_client.ml. It isn't overly complicated, and definitely not overkill.

However, doing that is getting very close to violating "H2- The plugins should be usable independently of Dune."