Git dependency with parent file dependency

[endocrimes]

👋

I'm trying to build a project that depends on generated bindings for a WIT world (https://github.com/fermyon/spin/blob/main/crates/world/src/lib.rs).

When building with cargo build, things work as expected and everything compiles correctly

When building with crane in a nix-flake however, the build fails with:

error: failed to read file "/nix/store/hivcpp5isdhmzzgj0q7920axnq7fq3xq-vendor-carg....
       Caused by:
           No such file or directory (os error 2)
  --> /nix/store/hivcpp5isdhmzzgj0q7920axnq7fq....
    |
3  | / wasmtime::component::bindgen!({
4  | |     inline: r#"
5  | |     package fermyon:runtime
6  | |     world host {
...  |
12 | |     async: true
13 | | });
   | |__^
   |
   = note: this error originates in the macro `wasmtime::component::bindgen` (in Nightly builds, run with -Z macro-backtrace for more info)

Although you can apply a custom source filter to your own packages, i.e:

    witFilter = path: _type: builtins.match ".*wit$" path != null;
    witOrCargo = path: type:
      (witFilter path type) || (craneLib.filterCargoSources path type);

    commonArgs = {
      src = pkgs.lib.cleanSourceWith {
        src = craneLib.path ./.;
        filter = witOrCargo;
      };

I haven't been able to find an (obvious) way to apply custom source filtering to the git dependency (and I'm not really a rustacean, so I'm unsure if there's something I'm missing that could be passed to cargo to ensure this works consistently).

I'd appreciate any pointers 😅

Reproduction example: https://github.com/endocrimes/crane-build-error-example

[endocrimes]

It looks like cargo vendor (after doing some issue digging) also doesn't include the wit files - Is there likely a better way to specify the dependency that would include them? (or am I somewhat out of luck?)

[endocrimes]

Ok so it seems like my issue is because the wit files are in a parent dir in the workspace - Do I have any way to construct that manually? (or an escape hatch to preserve a whole repo a la cargo defaults?)

[ipetkov]

Hi @endocrimes thanks for the report! I'm assuming you've probably come across #271 (comment) since your analysis about the WIT files being in the parent directory being the problem is spot on.

IMO this should be considered (and reported) as an upstream bug since this sort of pattern would fail to build with both cargo vendor and if the package was ever published to crates.io. (tl;dr cargo happens to check out entire repo for git dependencies and a crate peeking at files outside of its own directory happens to work there, which allows for bugs like these to go "unnoticed". but "normally" published crates can't really rely in files in their repo's parent directory since cargo wouldn't package that source when publishing the crate)

---

Do I have any way to construct that manually? (or an escape hatch to preserve a whole repo a la cargo defaults?)

Three options come to mind, listed in order of ease to implement:

1. Fork the repo and use a source override to point to your own fork with the issue fixed (hopefully until a solution is accepted upstream
2. Configure a Nix derivation which downloads the upstream sources and applies a patch with a fix. Then add a preConfigure hook to your actual package definition to apply a source override for the crate but using a path to the patched sources in the Nix store. This is basically the same as 1 but it avoids having to host an actual fork (at the cost of more complexity...)
3. If you are truly brave (and/or bored) enough to futz around with crane's vendoring internals there might be a way to achieve this, but it will not be easy or ergonomic so I don't recommend it (but happy to share any insights/guidance on how it works if you really want to go down that path)

[musjj]

@ipetkov
Is there instead a way to hook into a specific git dependency and modify the repository right after it was cloned (before the packaging occurs)?

[ipetkov]

@musjj We don't have a first class way of doing that atm, though you may be able to apply an override for downloadCargoPackageFromGit

[ipetkov]

With #615 we will have a general purpose mechanism for overriding (patching) any arbitrary crate sources!

Hopefully it should make it easier to work around little quirks like moving files around before they are vendored into their own separate directories

[ipetkov]

Here's a quick start snippet for how to patch dependencies: https://crane.dev/patching_dependency_sources.html