Dune #158

pqwy · 2019-04-03T02:28:42Z

Once this goes in, there is no cross-compilation for mirage. What is the current story? @avsm

avsm · 2019-04-03T10:53:49Z

Awesome! I'll take a look at the Mirage end later this afternoon, and review the dune changes as well.

hannesm · 2019-04-03T11:46:06Z

@pqwy AFAICT the current "mirage+dune cross" story is (as shown in https://github.com/inhabitedtype/bigstringaf - this will hopefully be less horrible soon)

META.package.template to embed the ocamlfind attributes (xen/freestanding_linkopts)
src/xen and src/freestanding subdirectories that (a) copy C files around, and (b) call out to a shell script to collect CFLAGS via pkg-config
optional dependencies in opam (ocaml-freestanding, mirage-xen-posix)

avsm · 2019-04-03T14:58:39Z

nocrypto.opam

+
+depends: [
+  "ocaml" {>= "4.03.0"}
+  "dune" {build}


Suggested change

"dune" {build}

"dune" {build & >="1.7"}

to match the dune version

Added, but I'm not sure what is the minimal dune language version that this needs?

avsm · 2019-04-03T14:59:04Z

nocrypto.opam

+  "ocplib-endian"
+  "zarith"
+  ("mirage-no-xen" | ("mirage-xen" & "zarith-xen"))
+  ("mirage-no-solo5" | ("mirage-solo5" & "zarith-freestanding"))


are these still needed since the mirage packages are removed from this repo?

The mirage package, nocrypto.mirage, is still present. It's just that it's meaningless because the stubs are not cross-compiled.

Hence the PR: the state of this branch is still incoherent.

avsm · 2019-04-03T14:59:33Z

@hannes I'm looking at the new cross-compilation story for this, so that we can keep the mirage-specific build pieces out of the mainline nocrypto repository and only use it to generate nocrypto-freestanding, nocrypto-xen, etc. I'm just working on a tree to demonstrate this (and reply to your mirageos-devel email), so I'll update as soon as I have that (hopefully tomorrow).

This PR looks good to me to merge and I can send any tweaks in a followup one.

pqwy · 2019-04-03T16:12:45Z

@avsm I cannot merge this as it is, because it immediately breaks mirage compatibility.

In general, I would strongly prefer a DRY solution, of the sort that I lost giving up ocamlbuild.

You suggestion is to remove mirage-related bits, and re-add them in a separate repository, correct? If this is the case, that repo needs to go up before this is merged, so that I can review the exact approach before proceeding.

emillon · 2019-04-09T13:45:24Z

config/cfg.ml

+  let fs = match Sys.getenv evar with
+    "true" -> flags
+  | "false" -> []
+  | _ -> auto


I think that this should be an error (or at least a warning)

Where's the error?

... right, if you set it but not to true or false. I was hoping I'd snatch a parser for configuration things from somewhere else, before I start writing that in the discovery tool.

I did try to emit text to confirm override, but dune seems to swallow my stderr.

emillon · 2019-04-09T13:46:10Z

dune-project

@@ -0,0 +1,3 @@
+(lang dune 1.7)
+(name nocrypto)
+(version %%VERSION_NUM%%)


I don't think that this is necessary

Suggested change

(version %%VERSION_NUM%%)

(version %%VERSION_NUM%%)

Correct, it's not strictly necessary. But it adds version to META files.

META files are automatically generated with the correct version. For example, ̀cstructdoesn't have(version)` here, and you can check that the installed META file has the correct version.

This is not what I'm seeing while playing with a pinned package. I won't pretend to understand why.

emillon · 2019-04-09T13:46:50Z

lwt/nocrypto_entropy_lwt.ml


 let some x = Some x

+[@@@ocaml.warning "-3"]


If that's for Lwt_sequence it's now possible to use Lwt_dllist.

Or more precisely, Lwt_main.Enter_iter_hooks. This is not the changeset for that.

TheLortex · 2019-04-10T12:21:42Z

Regarding mirage compatibility, I've done some experiments and it's possible to separate the nocrypto core and mirage packages in several repositories.

I've done the modifications here:

https://github.com/TheLortex/ocaml-nocrypto/tree/dune-separate-mirage contains the core of nocrypto
https://github.com/TheLortex/nocrypto-mirage contains mirage implementations

What changed in nocrypto is:

nocrypto is a virtual library that exposes the nocrypto.mli interface
nocrypto.default is the default implementation that compiles on unix
C and ml source files are installed along with the library
This allows nocrypto-mirage to grab these sources and recompile them using other flags.

The problem, for now, is that it's a 'file-by-file' procedure, meaning that we can't export and re-import a whole directory, so the work remains tedious.

pqwy · 2019-04-10T20:22:23Z

I'll be honest -- this falls a little short of looking impeccable.

It seems that there is a choice between exporting the build internals of a library to Mirage, versus exporting the build internals of Mirage to a library.

Is it not possible to simply add build flags for particular tools to an invocation of dune, and have it apply them to all recursive builds? Provided mirage-the-tool sets them, one could vendor in all the deps of a unikernel into a single directory and build everything correctly without either cooperation from third-party libs or any knowledge of those.

Because the only problem here is a limited form of cross-compilation: the C compiler needs to be given certain flags, the results need to be stored somewhere, and this somewhere needs to play nicely with the final step of linking the unikernel.

TheLortex · 2019-04-10T20:55:06Z

I definitely agree with you, this was to show that it's possible but definitely not good looking.

Is it not possible to simply add build flags for particular tools to an invocation of dune, and have it apply them to all recursive builds?

It's possible using different build contexts with different flags set, but I'm not sure the tooling is ready for that yet.

pqwy · 2019-04-10T21:50:05Z

If I'm not mistaken, as long as dune can be convinced to add flags (merely c_flags in this case) to an entire build, all that's left is to change stuff around here to prepend all of the _build/<$mirage_profile>/** to the linkflags (and remove the previous hack of querying ocamlfind for the alternative c library name).

Assuming

it's OK to use the working directory to store all of the build artefacts (and therefore recompile the deps when initially building a new unikernel); and
everything in Mirage is using dune,

is there anything that makes this scheme infeasible?

It would answer Adam Steen's question on the list from yesterday and make builds generally more controllable.

TheLortex · 2019-05-22T13:57:10Z

So it looks like we're going with what you suggested, and therefore it should be good for nocrypto to switch to dune. However we need some changes to get it working:

Use :standard in C flags to import workspace flags
Use bigarray-compat library to drop the unix dependency on ocaml 4.07
The diff is very small: TheLortex@4548e4e

One more thing I noticed is that nocrypto gets rid of sexplib but ocaml-tls still assumes that converters are available (https://github.com/mirleft/ocaml-tls/blob/master/lib/ciphersuite.ml#L44).
What is the plan for that ?

avsm · 2019-05-23T14:25:55Z

I think @hannesm generally wants to reduce his dependency on ppx, so removing sexp derivers from tls is likely the way to go

TheLortex · 2019-05-27T14:29:13Z

Well the problem is that tls is full of sexp derivers and we can't totally remove them as it's part of the public API (to pass client/server configuration). So this needs a refactor that is out of my scope I guess

no pack

9d46d4c

pqwy force-pushed the dune branch from 8a228b8 to 558c347 Compare April 3, 2019 02:30

dbuenzli mentioned this pull request Apr 3, 2019

Compilation of nocrypto fails TOPKG_CONF_DEBUGGER_SUPPORT=true dbuenzli/topkg#134

Closed

avsm reviewed Apr 3, 2019

View reviewed changes

pqwy force-pushed the dune branch from 558c347 to 9413cee Compare April 3, 2019 16:14

dune'd

80b7b4b

pqwy force-pushed the dune branch from 9413cee to 80b7b4b Compare April 3, 2019 23:27

emillon reviewed Apr 9, 2019

View reviewed changes

hannesm mentioned this pull request Apr 13, 2019

Use Dune instead of OCamlbuild to build unikernels mirage/mirage#969

Closed

psafont mentioned this pull request May 22, 2019

Update cstruct to 4.0 and its dependents xapi-project/xs-opam#387

Merged

avsm mentioned this pull request May 23, 2019

Windows fixes #162

Open

dinosaure mentioned this pull request Dec 16, 2019

Dunification of MirageOS mirage/mirage#1024

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dune #158

Dune #158

pqwy commented Apr 3, 2019

avsm commented Apr 3, 2019

hannesm commented Apr 3, 2019

avsm Apr 3, 2019

pqwy Apr 3, 2019

avsm Apr 3, 2019

pqwy Apr 3, 2019

avsm commented Apr 3, 2019

pqwy commented Apr 3, 2019

emillon Apr 9, 2019

pqwy Apr 9, 2019

pqwy Apr 9, 2019

emillon Apr 9, 2019

pqwy Apr 9, 2019

emillon Apr 10, 2019

pqwy Apr 10, 2019

emillon Apr 9, 2019

pqwy Apr 9, 2019

TheLortex commented Apr 10, 2019

pqwy commented Apr 10, 2019

TheLortex commented Apr 10, 2019

pqwy commented Apr 10, 2019 •

edited

TheLortex commented May 22, 2019

avsm commented May 23, 2019

TheLortex commented May 27, 2019

Dune #158

Are you sure you want to change the base?

Dune #158

Conversation

pqwy commented Apr 3, 2019

avsm commented Apr 3, 2019

hannesm commented Apr 3, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

avsm commented Apr 3, 2019

pqwy commented Apr 3, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TheLortex commented Apr 10, 2019

pqwy commented Apr 10, 2019

TheLortex commented Apr 10, 2019

pqwy commented Apr 10, 2019 • edited

TheLortex commented May 22, 2019

avsm commented May 23, 2019

TheLortex commented May 27, 2019

pqwy commented Apr 10, 2019 •

edited