Improve repodata.json download for proxy mirror mode

[beenje]

On our Quetz instance, we defined conda-forge as proxy channel.
We use local storage.

The repodata.json file from conda-forge is cached locally for channel.ttl seconds.
When the file is requested, if the time is older, the file is first downloaded locally. This is blocking and all clients requesting that file have to wait. It can take some time and we even experience some timeout on the client side: requests.exceptions.HTTPError: 504 Server Error: Gateway Time-out for url https://conda.example.org/get/conda-forge/linux-64/repodata.json.

I'm sure this could be improved. Just some ideas:

• the repodata.json file could maybe be downloaded in the background so that clients don't have to wait for the download.
• downloading repodata.json.zst if available should be faster?
• even better: use repodata.jlap incremental repodata to avoid downloading the full repodata.json every time. I read about it in https://conda.discourse.group/t/how-we-reduced-condas-index-fetch-bandwidth-by-99/257