Architecture - Toward cargo2nix 1.0

[psionic-k]

Generate Nix Expressions vs Use Cargo.lock Directly

As it is, Cargo.nix contains basically a transliteration of Cargo.lock into nix expressions with added feature conditions. Almost nobody uses rootFeatures AFAIK. Building each dependency independently requires this information, so it probably needs to be added to the Cargo.lock file. It may make more sense to adopt the Naersk & Crane approach, using the lock file directly. We can still know all of the inputs at evaluation time. We can still build every crate independently.

Independent Dependency Building is 100% Never Changing

In 2036, if 90% of your system programs are written in Rust, there is zero, zero chance that we build Rust applications by treating each workspace as if its dependencies are one big derivation instead of a bunch of tiny derivations. Tiny derivations, granular per crate, are absolutely the correct way. It's the only way to have any kind of cache efficiency, which becomes more and more important as your workspaces multiply. As systems become more and more Rust, sharing dependencies will become necessary for basic viability.

I'll leave alone the topic of ensuring identical derivations for those created from a Cargo.lock and those from nixpkgs, but if one of the 3rd party overlays takes over providing Rust in nixpkgs, such compatibility and cache sharing could be a realistic outcome.

Build-time Rust Helper Program (Proposal)

This is about the mkcrate.nix (and mkcrate-utils.sh) functionality that is the build phase. This stuff basically glues cargo into Nix.

This is the most fragile place in cargo2nix IMO. We use jq, remarshal, and a lot of bash. There is even a hack using sed to protect remarshal from certain valid toml. This is crazy. We are doing all of this so that cargo need not know nor understand that it is being used to build crates in isolation.

What if cargo did know that it was being used to build crates in isolation?

There is a perfectly good road towards this situation. If cargo2nix stops concerning itself with processing the Cargo.toml into nix expressions and instead focuses on replacing the big ball of bash & stdenv tools, we can make something in Rust that would actually be valid to include into cargo itself.

Likely need some support from Cargo team

Currently I don't think Cargo is explicitly supporting the Nix style build case. While the machine output is useful, if Cargo were aware of this case, it would have options to generate outputs that it can consume directly in the next build phase. This is generic to all Nix or other reproducible tools. So if we do this, it's the blueprint for cargo like tools from other languages.

[psionic-k]

Nixpkgs Overlay

Cache sharing is going to be an effective way to drive many decisions so that Rust & Nix benefit each other. In the ideal state:

• Rust crates can be expressed in nixpkgs in a way that dependents can articulate them as needed (platform, target, features, linking, native libs)
• Tools to generate expressions (like cargo2nix) intentionally create derivations that overlap with those in nixpkgs so that many crates are picked up from the hot nixpkgs caches
• Maintenance of a nixpkgs crate expression or a project being nixified as a whole both use similar workflows

The incremental steps to get there:

• cargo2nix overlay can interoperate with nixpkgs Rust toolchain or nixpkgs inherits it's Rust toolchain from an overlay that is good
• mkCrate is stabilized and supplied to nixpkgs. We only need mkCrate. We do not need workspaces or packageFun handling. If a crate can be built in isolation and expose enough information to dependents and consume it within those dependents' build phases, then multiple approaches to handling the workspace will have everything they need and can share derivation caching without knowledge of their workspaces.
• mkCrate dependencies are shifted to Rust tooling, making it possible for Cargo to learn these behaviors and later reduce the buildPhase dependencies to just a Rust toolchain

Some Important Consequences

Right now, only rootFeatures is expressed in Cargo.nix. This needs to be more generic. Crates might have their own features articulated, and the workspace style of cargo2nix isn't appropriate for that. The derivations that result from the independent Nix expressions may be common, but both workspace and individual crate style expressions need support in mkCrate

[psionic-k]

Dream2nix makes the same distinction between translators & builders. mkcrate.nix is basically the cargo2nix builder. The cargo2nix binary is the translator.

De-tangling mkcrate.nix

Currently the structure is a bit of a jumble of several potential operations. Some of the conditionals should be pulled out before they start overlapping too much. It will become impossible to keep track of what situation is being implemented or to decipher the procedure for a particular state if it's not detangled. #228