Accepting 429 status codes via Lychee github action arguments

[agriyakhetarpal]

Hi there! I am opening this after following up on previous issues and discussions with getting rate limited due to GitHub links (#644, #138 in lychee-acion) and tried the steps mentioned in the docs regarding rate limits.

As of now, allowing 429 seems like the best option for us. Is there a way to properly accept both 200 and 429 status codes? I have not tried using a config file yet considering args: --accept 200,429 should ideally work within the .yml file. Here is a link to the PR to help debug the issue: pybamm-team/PyBaMM#2734 and a link to the workflow file.

Currently, --accept 200,429, --accept [200, 429], and --accept "200,429" still fail on the github links. It is also to be noted that the action initially did set up and use a cache in previous runs but not in subsequent runs, although runs did not cache any github links (I am assuming that was because the links need to pass at least once before getting cached?)

I'd greatly appreciate any help on this with possible workarounds or if there's something that we're doing wrong. Thanks!

[mre]

Hi @agriyakhetarpal,

Is there a way to properly accept both 200 and 429 status codes?

Yes, as you said it's --accept 200,429.

For example, consider this call:

echo 'https://httpstat.us/200 https://httpstat.us/429' | lychee --verbose --accept 200 --max-retries 0 -
✔ [200] https://httpstat.us/200
✗ [429] https://httpstat.us/429 | Failed: Network error: Too Many Requests

(We read from stdin in this example.)

As you can see 429 fails.

Now if we were to accept 429 as an accepted status code, it would work:

echo 'https://httpstat.us/200 https://httpstat.us/429' | lychee --verbose --accept 200,429 --max-retries 0 -
✔ [200] https://httpstat.us/200
✔ [429] https://httpstat.us/429 | OK (429 Too Many Requests): Too Many Requests

I saw that you use --accept [200,429] in your action config. You might want to change that.

But in a previous run you actually used the correct syntax, so now I'm a little confused why it doesn't work for you. 🤔
Checked other people's pipelines and they don't seem to run into this issue: https://github.com/search?q=lychee-action+accept&type=code

For example here:
https://github.com/CollabCodeTech/backend-challenges/blob/a991e892ec74302e23e725fa45f521d04adb16f3/.github/workflows/links_checker.yml#L34-L41

It is also to be noted that the action initially did set up and use a cache in previous runs but not in subsequent runs, although runs did not cache any github links (I am assuming that was because the links need to pass at least once before getting cached?)

Actually 429 should be cached. Try to run the following commands:

echo 'https://httpstat.us/200 https://httpstat.us/429' | lychee --verbose --accept=200,429 --max-retries 0 --cache -
echo 'https://httpstat.us/200 https://httpstat.us/429' | lychee --verbose --accept=200,429 --max-retries 0 --cache -

A .lycheecache should be created after the first call and the second call should be much faster.

Your usage of the cache looks fine to me. I'll take a closer look at your pipeline and play around with the repo a bit to see if I can find the mistake.

[mre]

The cache only gets created if the pipeline succeeds. So in your case the remaining errors cause it from getting created. You could either set fail: false to create the cache file even in case of error, or wait until the issue with 429 is resolved.

Another thing I'm wondering about right now is if we should cache 429. At the end of the day, this might be a transient error, which could be resolved on a subsequent request.

[mre]

I found the reason for you failing CI pipeline:
The args were inside the env block, causing all arguments to be ignored and the default arguments to be used instead since the very beginning.

Provided a fix in agriyakhetarpal/PyBaMM#1 if you want to have a look.

[agriyakhetarpal]

Nice catch, thank you very much! The pipeline still fails due to other links which will be fixed. Please feel free to close this.

[mre]

Glad it helped.
Since this is a discussion and not an issue, I'm guessing we don't have to close it or lock the discussion. 😉

[mre]

Actually, this just in: https://github.blog/changelog/2023-03-14-adding-the-ability-to-close-github-discussions/ 😆
So we can indeed close the discussion as resolved now.

[agriyakhetarpal]

Oh, that's great! Yes, please close it