add an extra_download_args field to pass extra args to the download functions

[andyscott]

Adds an extra_download_args field to pass arbitrary args to the underlying download functions.

This makes it possible to download a wider variety of artifacts. For example, here's a manifest to download nickel from Homebrew's binary cache:

{
  "nickel": {
    "$schema": "https://raw.githubusercontent.com/theoremlp/rules_multitool/main/lockfile.schema.json",
    "__version": "1.5.0",
    "binaries": [
      {
        "kind": "archive",
        "os": "macos",
        "cpu": "x86_64",
        "url": "https://ghcr.io/v2/homebrew/core/nickel/blobs/sha256:d59d317f7d3602c9350d48ee825119e68faeaeb534a7fdc3063c187d3355a5d0",
        "file": "nickel/1.5.0/bin/nickel",
        "sha256": "d59d317f7d3602c9350d48ee825119e68faeaeb534a7fdc3063c187d3355a5d0",
        "extra_download_args": {
          "headers": {
            "Authorization": "Bearer QQ=="
          },
          "type": "tar.gz"
        }
      },
      {
        "kind": "archive",
        "os": "macos",
        "cpu": "arm64",
        "url": "https://ghcr.io/v2/homebrew/core/nickel/blobs/sha256:adfbdbc42f9fb5aefd181ea0a798e224a64bb71a46cba200a07a9ae87a744cba",
        "file": "nickel/1.5.0/bin/nickel",
        "sha256": "adfbdbc42f9fb5aefd181ea0a798e224a64bb71a46cba200a07a9ae87a744cba",
        "extra_download_args": {
          "headers": {
            "Authorization": "Bearer QQ=="
          },
          "type": "tar.gz"
        }
      },
      {
        "kind": "archive",
        "os": "linux",
        "cpu": "x86_64",
        "url": "https://ghcr.io/v2/homebrew/core/nickel/blobs/sha256:5ff7a7f4adbbb072d382de788b365d1c902b5eef093a7222ff386c89cc6bb296",
        "file": "nickel/1.5.0/bin/nickel",
        "sha256": "5ff7a7f4adbbb072d382de788b365d1c902b5eef093a7222ff386c89cc6bb296",
        "extra_download_args": {
          "headers": {
            "Authorization": "Bearer QQ=="
          },
          "type": "tar.gz"
        }
      }
    ]
  }
}

[mark-thm]

Thanks! We’d prefer to keep the lockfile pretty strict, would you mind reworking this to take just an explicit headers arg?

[andyscott]

Sure. Same with the type parameter?

[mark-thm]

Ah actually type is only valid for archive and pkg so I think I might need to rehab the spec here to be a union. I’ll take a pass tomorrow to fix this up.

[andyscott]

As a heads up since you're going to be thinking about the schema: I was planning to open another PR to add add support for multiple binaries from a single archive. E.g. the envtest suite from https://kubebuilder-tools.storage.googleapis.com/kubebuilder-tools-1.27.1-darwin-amd64.tar.gz contains:

kubebuilder/bin/etcd
kubebuilder/bin/kube-apiserver
kubebuilder/bin/kubectl

[mark-thm]

So, weirdly, it I'm getting errors that headers isn't a valid keyword argument to pass to download or download_and_extract despite what the docs say.

[mark-thm]

with respect to multiple files -- I think it'd be a bit tricky to support -- you won't end up downloading the same archive more than once as far as I can tell, but you would end up extracting it a few times on disk

[mark-thm]

superceded by #22 -- thanks for your original proposal