Can ktra be used as a transparent caching proxy for crates.io?

[astraw]

Hi, our internal CI system seems recently to be getting a number of failed downloads (see below for an example) from crates.io and anyway it's not very efficient to make lots of requests to crates.io for downloading the same crate again and again. Therefore, I was searching for a "transparent" caching proxy for crates.io and came upon ktra. I wonder if it can serve my envisioned purpose - it seems quite likely but I'd like to ask here, also to get any tips and perhaps to suggest updating the documentation with this use case in mind.

It seems like ktra may be able to perfectly fit my desired use case of a transparent caching proxy for crates.io:

1. I configure cargo inside the CI system to access our instance of ktra and thus only ktra downloads the packages from crates.io, which it does only once.
2. As we don't plan to host any private packages we would ideally not have to change the repositories (e.g. the Cargo.toml files) of the packages being built. Rather we would set this up in a way where it is only used for builds inside the CI system. So ktra would be "transparent" in that sense.
3. As a further idea, perhaps ktra could delete its mirrored cache of files once per day (or week or month) to avoid accumulating gigabytes of packages which are never used. I guess ktra could alternatively track number of downloads and delete any package from the crates.io mirror which hasn't been downloaded for some specified period of time.

If this is currently possible with ktra, I would be happy to hear it and receive any pointers about how to set it up. I could try to improve this for inclusion into the book if we end up going this route.

Alternatively, I imagine this may be the domain of a traditional, "standard" HTTP caching proxy like squid. At least, perhaps it would have previously been easy before everything was HTTPS. In that case, I suppose that ktra is overkill. Nevertheless, it would be useful for people like me to know how to set that up. So a pointer for that would still be greatly appreciated.

(Also, as I write this, I have discovered that others have similar issues and suggest reverting to an older nightly may solve the errors in CI builds. Nevertheless, for efficiency's sake, the idea of caching is still interesting, even if the specific errors I am seeing can be solved with using a different version of nightly rust and cargo.)

Finally, here is an example of an error we get with crates.io (This is not bug a bug with ktra. So far we are not useing ktra.):

$ cargo build
 Downloading crates ...
  Downloaded rand_chacha v0.1.1
error: failed to download from `https://crates.io/api/v1/crates/half/1.7.1/download`
Caused by:
  [55] Failed sending data to the peer (Connection died, tried 5 times before giving up)

[vi]

Yes, but you need to additional steps for it:

1. Clone https://github.com/rust-lang/crates.io-index to your repository;
2. Override download URL in config.json to http://127.0.0.1:8000/ktra/api/v1/mirror (or similar);
3. Add something like this to ~/.cargo/config:

[source.crates-io]
replace-with = "mirror"

[source.mirror]
registry = "file:///path/to/your/git/mirror"

Obviously, it can be used remotely as well, not just locally.