libfetchers/git: Allow Git Remote Helpers

[lorenzleutgeb]

Motivation

Use Nix with repositories that are (only) accessible via remote helpers, which are programs that are invoked by Git "under the hood" based on URL schemes.

This opens up lots of possibilities to interoperate with other storage/transport mechanisms.

Example remote helpers (by URL scheme, alphabetically):

• codecommit: aws/git-remote-codecommit
• gitopa: see docs.gitopia.com
• grave: rovaughn/git-remote-grave
• hg: glandium/git-cinnabar
• ipfs: cryptix/git-remote-ipfs, dhappy/git-remote-ipfs
• ipld: ipfs-shipyard/git-remote-ipld
• mango: axic/git-remote-mango
• rad: see radicle-dev/heartwood and radicle.xyz/guides/user
• rclone: datalad/git-remote-rclone
• ssb: clehner/git-ssb

Context

The following two changes (both part of this PR) make use of such remote helpers possible in conjunction with Nix:

1. Relax URL scheme filtering for the Git fetcher, such that unknown URL schemes are not rejected directly. In case there is no corresponding remote helper available, the following output is produced:

   warning: URL scheme 'git+invalid' is non-standard and requires 'git-remote-invalid'.
   git: 'remote-invalid' is not a git command. See 'git --help'.
   error: program 'git' failed with exit code 128
   

2. Add GIT_DIR to the environment, since this is required by remote helpers.

As submitted (ee66ec3) this is a proof of concept, and works for me locally. I do understand that this feature might have to be protected by an experimental flag and docs would have to be added/changed. I am happy to do this if there's positive feedback.

Priorities and Process

Add 👍 to pull requests you find important.

The Nix maintainer team uses a GitHub project board to schedule and track reviews.

[spacefrogg]

I want to raise the following question / thought: This is a quite general approach. Can it be expected that all remote helpers can be called without any additional command-line options? And if not, would they better be supplied when calling nix, i.e.,

nix ... --remote-helper-opts '--a-opt a-val' git+helper://...
# or
NIX_REMOTE_HELPER_OPTS=... nix git+helper://...

or via the URL to properly pair the options with the remote helper call (for nested fetches of flake inputs etc.)?

nix ... 'git+helper://...?opts=--a-opt%20a-val'

[lorenzleutgeb]

I want to raise the following question / thought: This is a quite general approach. Can it be expected that all remote helpers can be called without any additional command-line options?

Yes. Remote Helpers are invoked by Git and the Git reference documentation on Remote Helpers states:

Remote helper programs are invoked with one or (optionally) two arguments.
The first argument specifies a remote repository as in Git; it is either the name of a configured remote or a URL.
The second argument specifies a URL; it is usually of the form <transport>://<address>, but any arbitrary string is possible.

Regarding

[...] or [would they better be supplied] via the URL to properly pair the options with the remote helper call (for nested fetches of flake inputs etc.)?

nix ... 'git+helper://...?opts=--a-opt%20a-val'

This is something that Remote Helpers can leverage without Git passing arguments. They control the URL scheme, so during git clone 'git+example://x?a-opt=a-val', the remote helper git-remote-example is invoked with the argument example://x?a-opt=a-val and can interpret the URL.

However, it will be tricky to get a URL with query string past Nix, since it also uses query strings to encode fetching information. With a file called git-remote-example in $PATH that just writes its $@ to a file, I was able to verify that

$ nix ... 'git+example://test?x=y&ref=main'

results in

example://test

and conversely

$ nix ... 'git+example://test%3Fx=y?ref=main'

results in

example://test%3Fx=y

So, Remote Helpers that do this, and require syntax that is otherwise parsed by Nix, will be incompatible. One workaround is the one you mentioned, somehow prefixing arguments that should not be parsed by Nix, e.g. ?pass-x=y. One loses the ability to just copy the URLs that the protocol of the remote helper requires (and maybe add ?rev or so).

IMO the conflation of fetching arguments and URL are the issue here, but that's Pandora's box. I think that there still are some interesting use cases. I am successfully using Remote Helpers with this patch.