Extendable updating/fetching

[shajra]

Hi.

It occurred to me that there's a whole slew of "enhancement" fetching issues already in the tracker:

• Add support for Docker fetcher #77
• Add support for Hackage fetcher #78
• Add support for npm fetcher #79

A few observations:

• These only address fetching, but not updating.
• Extension would be nice in a general way to address experimental, niche, or corporate-internal needs.

As @michaelpj pointed out in the comments, there's some possibility for extension via a Nix expression (#137). But the logic for updating is still part of the Haskell code of Niv. It would be nice to see if we could factor out updating to non-Haskell as well.

I think fetching is more obviously solved with a Nix expression, because fetching in the Nix work is a pure function.

Updating, is a bit more tricky because of side-effecting, specifically reaching out to the internet and getting an answer that might change from moment to moment. But one of the things that Nix can produce is not only a function, but a runCommand-style script. This script can function like IO in the Haskell world. And I think that gives us the full flexibility we need to make a Nix-based extension system that covers both fetching and updating.

This approach to extension would be similar to what Direnv does. Like Direnv, you could continue to have built-in support for the things that the most users will benefit from. But also, by having community support with these extensions, you will also get some inspiration for what's most useful, and how to implement it. One can feed the other.

What do you think?

[michaelpj]

I still think the approach I suggested in #137 was nice for this kind of customization. You feed sources.nix an overlay when you import it and you can use that to define extra fetchers.

[shajra]

@michaelpj I scanned the PR there. It's good to know it's there. I missed that feature.

From what I saw, this seems to address fetching, but not updating, right? I think in my original issue, I didn't make clear that updating is something that I'm really interested in solving for as well, because it's a large portion of what I find Niv useful for. Otherwise, I'm just crawling a lot of websites.

But if your approach covers updating as well, then I'm pretty happy. Just need to play around with it and see how it works.

[michaelpj]

Yes, you're quite right, it only handled fetching! At the moment I think all the updating is done by the Haskell code, which makes it a bit harder to extend in an ad-hoc way.

[michaelpj]

Also, to be clear, that was just a suggestion, it's not actually implemented in niv.

[shajra]

Yes, you're quite right, it only handled fetching! At the moment I think all the updating is done by the Haskell code, which makes it a bit harder to extend in an ad-hoc way.

Okay, I think we're converging on the same sentiments. One reason to use a shell breakout is because it allows us to do just a little bit more... specifically the stuff that's in Haskell for updating.

But I don't want to completely cut Haskell out of the equation. I think for the common cases, it's perfectly reasonable to not have to call shell at all. Shell would just be an alternate path.

I'll update the description of this issue to make clear my interest in updating in addition to fetching.

[shajra]

@michaelpj since you're involved in the Haskell.nix world, you may appreciate that the interest in this ticket butts up against my interest in better automating something like the management of materializations for a Haskell.nix project. The materialization scripts are similar in spirit to the scripts I proposed in this issue's description. Except... Haskell.nix scripts actually modify the filesystem. I was thinking originally of scripts that didn't modify the filesystem, but output JSON to then be injected back into sources.json (possibly with some transformation). Also, Haskell.nix is kind of two-stage (I think)... one to get the hash of the new materialization, and another to save it to the filesystem.

Ultimately, I want happy-path, push-button updating of Nix projects, with good error reporting when things go wrong. At that point, have I gone way past the original purview of Niv? Am I really going done that path of making a different tool entirely that may subsume some of the functionality of Niv?

[shajra]

I was also reminded that @expipiplus1 also maintains an application that's in this space... https://github.com/expipiplus1/update-nix-fetchgit

I feel that the approach there of syntactically matching Nix expressions might be too fragile for what I have in mind, though. I can appreciate the appeal of hitting the 80% case. I just find myself in the 20% case a lot more with my personal projects (Haskell.nix being one of those situations for me).

[colemickens]

I also would like this, and is part of the reason that I cannot adopt niv for nixpkgs-wayland. nixpkgs-wayland contains a script, update.shthat usesnix-prefetch` to update packages sources and hashes (including cargoSha256 and vendorSha256) in a fully automated, CI-driven fashion.

If niv supported extensible fetching/updating, then I could probably wire up nix-prefetch and then use niv to drive most of the updating process.