Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle custom registries #214

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

Anderssorby
Copy link
Collaborator

To fix #186

  • Load cargo config and control url in unpackCrate

@blitz
Copy link
Member

blitz commented Dec 17, 2021

@Anderssorby Looks pretty uncontroversial. Is there something in the test suite that tests this?

Copy link
Collaborator

@nmattia nmattia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but I'm a bit rusty on how everything here works, so letting @blitz approve officially!

jonringer
jonringer previously approved these changes Jan 7, 2022
lib.nix Outdated Show resolved Hide resolved
@jonringer
Copy link

I would say if the old workflow is still correct, then we should merge. At a glance, I don't see anything wrong. If something is wrong, then someone else can create an issue, and provide feedback as to how it's now working in their use case.

@Anderssorby do you mind squashing the fixup commit?

Update lib.nix

Co-authored-by: Jonathan Ringer <jonringer@users.noreply.github.com>
@Anderssorby
Copy link
Collaborator Author

Anderssorby commented Feb 14, 2022

@blitz So should we merge this?

Copy link
Member

@blitz blitz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@blitz So should we merge this?

The code looks good, but one of the "slow" tests (as opposed to fast tests we execute in the CI) are still failing:

$ nix-build test.nix --show-trace -A rustlings 
error: please specify either 'src' or 'root'

       … while evaluating 'resolveSource'

       at /home/julian/src/own/naersk/lib.nix:14:19:

           13|   # not defined.
           14|   resolveSource = attrs:
             |                   ^
           15|     let

       … from call site

       at /home/julian/src/own/naersk/default.nix:28:16:

           27|       # Config for cargo itself
           28|       source = libb.resolveSource arg;
             |                ^
           29|       configFile = source.root + "/.cargo/config.toml";

       … while evaluating 'optionalString'
```
   at /nix/store/vqd0zjq6qjs5zz2m1nnzp4w5glf7ga7r-nixpkgs-src/lib/strings.nix:202:5:

      201|     # String to return if condition is true
      202|     string: if cond then string else "";
         |     ^
      203|

   … from call site

   at /home/julian/src/own/naersk/lib.nix:228:15:

      227|             mkdir -p $out/.cargo
      228|             ${lib.optionalString (! isNull cargoConfigText) "cp ${config} $out/.cargo/config"}
         |               ^
      229|             cp ${cargolock'} $out/Cargo.lock

   … while evaluating the attribute 'buildCommand' of the derivation 'dummy-src'

   at /nix/store/vqd0zjq6qjs5zz2m1nnzp4w5glf7ga7r-nixpkgs-src/pkgs/stdenv/generic/make-derivation.nix:205:7:

      204|     // (lib.optionalAttrs (attrs ? name || (attrs ? pname && attrs ? version)) {
      205|       name =
         |       ^
      206|         let

   … while evaluating the attribute 'src' of the derivation 'rustlings-deps-1.3.0'

   at /nix/store/vqd0zjq6qjs5zz2m1nnzp4w5glf7ga7r-nixpkgs-src/pkgs/stdenv/generic/make-derivation.nix:205:7:

      204|     // (lib.optionalAttrs (attrs ? name || (attrs ? pname && attrs ? version)) {
      205|       name =
         |       ^
      206|         let

   … while evaluating the attribute 'builtDependencies' of the derivation 'rustlings-1.3.0'

   at /nix/store/vqd0zjq6qjs5zz2m1nnzp4w5glf7ga7r-nixpkgs-src/pkgs/stdenv/generic/make-derivation.nix:205:7:

      204|     // (lib.optionalAttrs (attrs ? name || (attrs ? pname && attrs ? version)) {
      205|       name =
         |       ^
      206|         let
</details>

@Patryk27
Copy link
Contributor

Patryk27 commented Apr 7, 2022

Hi, @Anderssorby - thanks for this merge request! Would you mind rebasing it & addressing the comment from @blitz?

@Anderssorby
Copy link
Collaborator Author

Sure, I can take a look later.

@SkamDart
Copy link

@Anderssorby Awesome work. I would also like to see this feature get merged in. Is there anything I can do to help you close out this PR?

@Frarese
Copy link
Contributor

Frarese commented Aug 24, 2022

Hello, I would also like to see this feature merged and I tried to understand why the test wasn't passing.
Apparently we check for the attributes root or src but in the case of rustling we pass the source like this:

rustlings = naersk.buildPackage sources.rustlings;

instead of this:

rustlings = naersk.buildPackage {
    src = sources.rustlings;
};

There are two ways to fix this issue:

  1. We keep the code as is and change the test to reflect that we always need to specify src or root;
  2. Or, as i did in this commit, if both attributes are missing we pass attrs itself as a source.

Let me know if I can contribute further ☺️

@Patryk27
Copy link
Contributor

Let me know if I can contribute further

Thanks for analyzing the issue, your second approach seems the most fitting - I'll try to rebase this pull request with your patch applied over the weekend 🙂

@Patryk27
Copy link
Contributor

Patryk27 commented Aug 27, 2022

Hmm, maybe I'm misunderstanding something, but the solution presented in this pull request can't work, can it? 👀

  • host, index and url as passed to unpackCrate are never really set anywhere (besides being assigned their default values),
  • We don't checkout the registry's index, which we need to do in order to load the registry's configuration (https://doc.rust-lang.org/cargo/reference/registries.html#index-format) - otherwise naersk can't possibly know where to download the package from.

So, say, we're given a Cargo.lock that says:

[[package]]
name = "dep"
version = "0.1.0"
source = "registry+https://git.some-company.com/rust/crates-index"
checksum = "bae3d8de1b7fd1fef6c2da3130a7d06d32499fd5292a9c1309681ac79e98c643"

To actually download that package naresk would have to:

  1. Checkout https://git.some-company.com/rust/crates-index,
  2. Load config.json from in there, parse its dl key,
  3. Call unpackCrate with the results of that.

... and I think the first step is fundamentally problematic in Nix, since we don't know at which commit we should checkout that hypothetical rust/crates-index (considering that checksum in lockfile refers to the package itself, not its registry's index).

I'd say that in order to have some minimum viable product we should extend buildPackage, so that instead of parsing .cargo/config.toml it simply accepts a new parameter describing registries required to build the package:

{ ... }:

naersk.buildPackage {
  src = ./.;
  
  registries = {
    # crates-io remains implied, ofc.
    
    custom = {
      dl = "https://my-registry.io/api/v1/crates/{version}/{crate}/whatever/{lowerprefix}";
    };
  };
}

I think this would solve all of the problems here, but I'm leaving this one up to discussion 🙂

Edit: I see that Crane uses a similar solution 😄 (https://github.com/ipetkov/crane/blob/5548a68f5d4d60a2315ab1232e6fba5528e71dc8/examples/alt-registry/flake.nix#L35)

@Frarese
Copy link
Contributor

Frarese commented Aug 30, 2022

So I started looking into it again and have an implementaton of buildPackge that looks like this:

buildPackage = args@{registries? {}, ... }:

that passes the fast tests but fails the slow ones:

Building
NOT testing on darwin, enabling sandbox
error: value is a string with context while a set was expected

       at /home/frarese/repos/naersk/test/slow/talent-plan/default.nix:12:19:

           11|
           12|   talent-plan-3 = naersk.buildPackage "${sources.talent-plan}/rust/projects/project-3";
             |                   ^
           13| }

should I just make a registriesCustomDl attribute so that we can maintain the signature for the function or is there a better way to include registries in the options for buildPackage ?

@Patryk27
Copy link
Contributor

To create a new parameter, you should just add it to build.nix - that will not only later include it in README.md, but also compose neatly with rest of the library 🙂

@secana
Copy link

secana commented Feb 25, 2024

Hi! What is missing to get this PR through? I'm the maintainer of https://kellnr.io and I would really like to be able to use a private registry for Rust with Nix. Unfortunately, I'm very new to Nix, so I lack the skill to implement the feature myself.

@Patryk27
Copy link
Contributor

Hi, @secana - considering Naersk doesn't have a maintainer at the moment, the first step would be finding/becoming one 😅

Realistically, I'd suggest switching to Crane, though - it's (imo) better designed and it's kept very much up to date in terms of support and features.

@nmattia
Copy link
Collaborator

nmattia commented Feb 26, 2024

I'd suggest switching to Crane, though - it's (imo) better designed and it's kept very much up to date in terms of support and features.

Oh wow, Crane looks nice! Since it's maintained, well documented and supports many more features I think it's best to archive naersk and point people to Crane; best if people don't try setting up naersk just to realize it's deprecated anyway.

Unless people complain I'll do that soon!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Private crate registries
8 participants