Error: input stream: Status code: 429 #444

Harmiox · 2019-07-09T23:17:23Z

Everything was working fine earlier today and yesterday until all of a sudden I started getting this error multiple time in my log file. I've never had a 429 response with this lib before. (Edit: This issue only happens on my VPS, it doesn't happen on my local machine.)

Running ytdl-core@0.29.3

import ytdl from 'ytdl-core';
const ytdlOptions: {} = { filter: 'audioonly', quality: 'highestaudio' };
ytdl('https://youtube.com/watch?v=O6RyKbcpBfw', ytdlOptions);

Error: input stream: Status code: 429
    at ClientRequest.httpLib.get (/root/discord/music/poco/node_modules/miniget/lib/index.js:125:19)
    at Object.onceWrapper (events.js:286:20)
    at ClientRequest.emit (events.js:198:13)
    at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:556:21)
    at HTTPParser.parserOnHeadersComplete (_http_common.js:109:17)
    at TLSSocket.socketOnData (_http_client.js:442:20)
    at TLSSocket.emit (events.js:198:13)
    at addChunk (_stream_readable.js:288:12)
    at readableAddChunk (_stream_readable.js:269:11)
    at TLSSocket.Readable.push (_stream_readable.js:224:10)

The text was updated successfully, but these errors were encountered:

encloinc · 2019-07-10T00:26:24Z

Same thing is happening to me!

encloinc · 2019-07-10T00:30:41Z

What seems to happen is, like you im using a vps, it stops working after like a few requests.

fent · 2019-07-10T01:08:09Z

this status means that too many requests are being made too quickly

https://httpstatuses.com/429

I recently updated miniget, the http library that this library uses. the reconnect logic was refactored and moved around. so at first i thought that might be the issue. but in #445, @encloinc is using ytdl-core@0.29.2, which uses the old version of miniget.

@encloinc, can you run npm ls miniget and see what version it prints?

could also be that youtube changed their website, adding more rate limiting.

encloinc · 2019-07-10T01:09:29Z

-- ytdl-core@0.29.2
  +-- m3u8stream@0.6.2
  | `-- miniget@1.5.1  deduped
  `-- miniget@1.5.1```

encloinc · 2019-07-10T01:10:12Z

I dont know if its a rate limiting issue though, because when I restart the vps it works again for a few requests. Its quite weird. (And I wouldnt be like surpassing the limit too much, my vps makes no more than a request every 3 minutes)

fent · 2019-07-10T01:15:12Z

if youtube is rate limiting requests more, one solution can be to

-- ytdl-core@0.29.2
+-- m3u8stream@0.6.2
| -- miniget@1.5.1 deduped -- miniget@1.5.1```

ok thanks. that means this issue is originating from a change from youtube, rather than the recent update.

I dont know if its a rate limiting issue though, because when I restart the vps it works again for a few requests.

it's working again because there was some time when your vps didn't make any requests, not because it was restarted.

And I wouldnt be like surpassing the limit too much, my vps makes no more than a request every 3 minutes

is this for a bot that's shared in a server? does it get a lot of requests?

encloinc · 2019-07-10T01:17:11Z

if youtube is rate limiting requests more, one solution can be to

-- ytdl-core@0.29.2
+-- m3u8stream@0.6.2
| -- miniget@1.5.1 deduped -- miniget@1.5.1```

ok thanks. that means this issue is originating from a change from youtube, rather than the recent update.

I dont know if its a rate limiting issue though, because when I restart the vps it works again for a few requests.

it's working again because there was some time when your vps didn't make any requests, not because it was restarted.

And I wouldnt be like surpassing the limit too much, my vps makes no more than a request every 3 minutes

is this for a bot that's shared in a server? does it get a lot of requests?

It is a youtube-mp3 converter site, and it doesnt really. Also when I restart the vps it restarts in like 5 minutes. All I do is restart the script.

HcgRandon · 2019-07-10T01:33:31Z

Same here, And I'm on an older version. Def a YouTube change. Hopefully there is a workaround

encloinc · 2019-07-10T01:34:19Z

Yeah.

edunad · 2019-07-10T07:14:28Z

Just started happening as well for me, i use this for a personal player and i'm the only user

encloinc · 2019-07-10T07:16:05Z

Yeah it definitley isnt related to actual rate limits I think.

mh4ck · 2019-07-10T08:37:49Z

Same here, we're fucked up.

HcgRandon · 2019-07-10T13:49:30Z

I have friends seeing some issues but much less then me, however they use Lavaplayer. Additionally they also have US servers, mine are EU, might just mean it's not deployed globally yet. Some very basic testing proves they are cutting off requests pretty quickly. I'm wondering if they are detecting ytdl-core specifically somehow. Or maybe the user agent etc. I'll be getting some sleep then testing this pretty extensively I'll keep you updated to my findings

Perflyst · 2019-07-10T15:31:55Z

Google/YouTube has blocked my whole /64 IPv6 subnet due to one IPv6 address which I deployed recently to my invidious instance. The IPv4 address worked since a few months and is still working. Only IPv6 is blocked.

What provider do you use? I use Hetzner. Maybe it is something provider specific?

odedhb · 2019-07-10T15:43:44Z

It looks like just a new, more strict, throttling.
I’m trying exponential backoff.

mh4ck · 2019-07-10T16:10:30Z

I added some request caching (espacially for the video info) to my code - now 60% of servers are up again.
Maybe interesting for somebody.

Harmiox · 2019-07-11T01:34:08Z

I'm still receiving this issue, and my program is barely even using any requests (maybe 50 In total today). Does anyone know if this is provider specific? I'm not getting this issue on my local machine at all, just on my VPS.

encloinc · 2019-07-11T02:27:36Z

I fixed it by deploying different instances of the actual part of the code that downloads the youtube mp3's and having the main server pipe into one of these instances (at random). I made 15 of them, this seems its the only way to get past this atm.

mh4ck · 2019-07-11T14:23:38Z

I fixed it by deploying different instances of the actual part of the code that downloads the youtube mp3's and having the main server pipe into one of these instances (at random). I made 15 of them, this seems its the only way to get past this atm.

You did it from 1 IP?

encloinc · 2019-07-11T16:07:11Z

I dont know if the service uses 1 ip, they do it with "deployments". Either way it seemed to fix it.

…

On Thu, Jul 11, 2019 at 9:23 AM m.h4ck ***@***.***> wrote: I fixed it by deploying different instances of the actual part of the code that downloads the youtube mp3's and having the main server pipe into one of these instances (at random). I made 15 of them, this seems its the only way to get past this atm. You did it from 1 IP? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#444?email_source=notifications&email_token=ADI2LIVATHASWBIGI6FSZFLP64665A5CNFSM4H7KBMAKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZW3S2Y#issuecomment-510507371>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADI2LIWPFUC7V5QXG7D5LJTP64665ANCNFSM4H7KBMAA> .

mh4ck · 2019-07-11T16:37:24Z

Anybody else had figured out smth. ?

Interesting questions are:
How much requests could be done before they ban the IP?
How long will it take till the IP's are unblocked?

Haven't figured it out now.

WaqasIbrahim · 2019-07-11T16:40:22Z

@mh4ck I used 2 proxies (not rotating, static ip) last night for two different instances that were both blocked . But they seemed to work fine and did not get blocked even with high parallel traffic and still seem to be working.

mh4ck · 2019-07-11T16:59:45Z

@mh4ck I used 2 proxies (not rotating, static ip) last night for two different instances that were both blocked . But they seemed to work fine and did not get blocked even with high parallel traffic and still seem to be working.

Where you got the proxies from? I bought today a lot of new ip's but all got blocked now.
Each IP had sent ~1 request in 10-30s...

WaqasIbrahim · 2019-07-11T17:37:14Z

@mh4ck It's a local service, so can't reveal that. But I have added more just to be safe and so far it has resolved the issue. What VPS service are you using? I tried adding another instacne and that was blocked instantly. Maybe it's targeting spefic vendors?

omarroth · 2019-07-11T22:01:31Z

See also ytdl-org/youtube-dl#21729.

Similar to what @Perflyst mentioned I believe this is only an issue when connecting to YouTube over IPv6.

If possible, you may try disabling it or force connecting over IPv4. Unfortunately I'm not sure how easy that is only using this library, but it may be possible to disable it depending on cloud provider or some other configuration.

mh4ck · 2019-07-12T11:44:31Z

See also ytdl-org/youtube-dl#21729.

Similar to what @Perflyst mentioned I believe this is only an issue when connecting to YouTube over IPv6.

If possible, you may try disabling it or force connecting over IPv4. Unfortunately I'm not sure how easy that is only using this library, but it may be possible to disable it depending on cloud provider or some other configuration.

I'm only using IPv4 and still got this error after a few requests.

ShivamJoker · 2019-07-12T12:17:53Z

Guys guys guys please fix this issue as fast as possible. I have worked so hard on my music app from months if this thing happens my all work will be wasted @fent
Getting this today just requested few songs and error 429 coming
using AWS EC2 Mumbai

mh4ck · 2019-07-12T12:31:35Z

Guys guys guys please fix this issue as fast as possible. I have worked so hard on my music app from months if this thing happens my all work will be wasted @fent
Getting this today just requested few songs and error 429 coming
using AWS EC2 Mumbai

I guess there is nothing programmatically to do.
YouTube just ban IP's that requests videos to fast.

ShivamJoker · 2019-07-12T12:35:22Z

@mh4ck so we are fucked up now. 😭
and how do i unban it

katsumi143 · 2019-09-02T13:01:45Z

This is happening to me too, It's pretty annoying for me and other people.
Anyone found a workaround yet?

Harmiox · 2019-09-07T02:12:50Z

I tried using another lib (LavaLink) but I still ran into the 429 issue.

WaqasIbrahim · 2019-09-13T06:45:59Z

@fent Can you give a brief explaination how fent/node-miniget@1ed8467 solves this issue? Does youtube send retry-after header? And how slow would it be when it's rate limited? Thanks

fent · 2019-09-14T00:39:59Z

I don't know if youtube does, because I was never able to replicate it. but even if they don't, miniget will still retry the request.

WaqasIbrahim · 2019-09-14T10:44:34Z

@fent Thanks.

Atruo · 2019-10-02T12:38:00Z

@fent so basically if I change this file it will work? fent/node-miniget@1ed8467

mahhov · 2020-08-06T21:26:23Z

This still occurs, and retrying doens't help since all requests begin returning 429. I'm not sure this should be closed.
For my use case, I sent 1 request per 3-5 minutes, for about 7 hours (so about 140 requests total).
I've sent 100 requests / hour for previously (months ago) and not had a similar issue.

edit, nevermind, I see this issue is still open here #635

fent · 2020-08-06T21:39:05Z

have you tried using cookies?

edit, nevermind, I see this issue is still open here #635

I've kept that opened to remind myself to add throttling

mahhov · 2020-08-06T22:36:49Z

Yes, cookies worked, thanks.
I'm not sure throttling alone will avoid the issue since my use case was already a pretty slow rate of requests (1 per 3-5 minutes).

mh4ck · 2020-08-06T22:49:05Z

How long the cookies work, when they expire? Or they don't? Did anyone got experience with that?

mahhov · 2020-08-06T22:54:43Z

I would guess they would work for a few months; i.e. however long your youtube login remains valid.
Maybe there's a way to retrieve a new cookie with oauth?

mh4ck · 2020-08-06T23:53:20Z

I just tried cookies again but can't get it to work. Do you use the same IP for the cookie wich you used to create the cookie?

I generated it on my local pc and used it on my server - now i get the following error:
**Error parsing info: Cookie header used in request, but unable to retrieve video metadata**

I'm pretty sure to use the right Cookie and identity-token... made xss attacks 10 years ago and normaly know what i'm doing haha...

let cookie = worker.getCookie();
        let idToken = worker.getIdentityToken();
        if(cookie && idToken) {
          console.log("[Info]: Using cookie for info request.");
          if(typeof opts.requestOptions.headers == "undefined") {
            opts.requestOptions.headers = {
              cookie: worker.getCookie(),
              'x-youtube-identity-token': worker.getIdentityToken()
            }
          } else {
            opts.requestOptions.headers.cookie = worker.getCookie();
            opts.requestOptions.headers['x-youtube-identity-token'] = worker.getIdentityToken();
          }
        }
ytdl.getInfo(videoId, opts)
        .then(handleResolve)
        .catch(handleReject);

mahhov · 2020-08-07T05:59:52Z

idk about your specific use case and what your worker is?
But for a simple example, just do:

ytdl(<id>, {
            ...
            requestOptions: {
                headers: {
                    'Cookie': '<paste cookie as string>'
                }
            }
        })

Where your cookie is a string (not an object) that includes SID, HSID, and SSID; e.g.
'SID=<~70 characters>.;HSID=<~15 chars>; SSID=<~15 chars>;' Note, unlike JSON, it's semicolon ; deliminated and uses = instead of :.

mahhov · 2020-08-07T06:02:03Z

I've also gotten it to work with oauth, which imo is more versatile since it would be awkward to ask users to copy & paste cookies from their browser (unless there's an automated way of retrieving cookies I don't know about?)

ytdl(<id>, {
            ...
            requestOptions: {
                headers: {
                   Authorization: "Bearer ...",
                   Content-Type: "application/json"
                }
            }
        })

fent · 2020-08-07T09:03:41Z

what's all you need for oauth? i'd be interested in adding another example to the repo

mahhov · 2020-08-07T14:19:28Z

I use oauth for 2 things in my app.

(unrelated to ytdl), I use oauth to allow the user to add & remove videos to their playlists, and
to avoid the 429 error when downloading videos with ytdl in lieu of asking the user to copy paste a cookies from their browser.

notadevps · 2020-08-17T02:57:22Z

I am also getting same error any solutions?

cchen028 · 2020-08-18T20:57:02Z

I'm getting 429 error as well. I hosted on google cloud function to get video info from Youtube using ytdl.GetBasicInfo, it seems like it happens every couple days, the last time it happened it made total of 9 requests before throwing the error. Though when I look at the log it seems like for every request it actually makes 2 request (one returning 204 preflight and one returning 200), not sure if this would be an issue? Anyone else facing similar issues?

mahhov · 2020-08-19T04:48:32Z

Did you guys (@notadevps & @cchen028) try using cookies or oauth token?

notadevps · 2020-08-19T14:09:25Z

yea i tried using cookies but it didn't worked

mahhov · 2020-08-19T16:31:18Z

Did your cookie look like: 'SID=<~70 characters>.;HSID=<~15 chars>; SSID=<~15 chars>;'? Do you have a minimal reproducable example?

fent · 2020-08-19T17:32:02Z

if you got your cookies from document.cookies, try getting them from the header list in a request in the network tab. I've been told document.cookie omits some cookies

cchen028 · 2020-08-20T00:05:00Z

Thanks for the prompt responses. Yes I have added the cookies and so far it is still working (Will give it a couple more days).

The reason I'm asking is because it works fine on my compute engine which runs 24/7 but not on the cloud functions. The 3 failures (err 429) I've encountered on cloud functions follow a similar pattern which is after the functions have been idled for a while and on the initial request it would fail until I re-deploy the function again. Just figured if anyone with a deeper understanding would know the potential cause behind this?

I'm also trying to keep the functions alive to see if this will solve it

TehPigYT · 2020-09-19T06:40:10Z

I'm getting the 429 status code too, sadly.
This is what I get in console

Error: input stream: Status code: 429
at ClientRequest.<anonymous> (/home/container/node_modules/miniget/dist/index.js:150:31)
at Object.onceWrapper (events.js:417:26)
at ClientRequest.emit (events.js:310:20)
at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:603:27)
at HTTPParser.parserOnHeadersComplete (_http_common.js:119:17)
at TLSSocket.socketOnData (_http_client.js:476:22)
at TLSSocket.emit (events.js:310:20)
at addChunk (_stream_readable.js:286:12)
at readableAddChunk (_stream_readable.js:268:9)
at TLSSocket.Readable.push (_stream_readable.js:209:10)

Are there any fixes for this?

fent mentioned this issue Jul 10, 2019

Status code 429 #445

Closed

moonstar-x mentioned this issue Sep 11, 2019

bot cant play anything moonstar-x/discord-music-24-7#2

Closed

omarroth mentioned this issue Sep 13, 2019

Proxy config option / only API mode and requests per minute iv-org/invidious#742

Closed

fent closed this as completed in fent/node-miniget@1ed8467 Sep 13, 2019

omarroth mentioned this issue Oct 9, 2019

Video unavailable iv-org/invidious#811

Closed

omarroth mentioned this issue Nov 12, 2019

Video Unavailable pytube/pytube#499

Closed

dashimaki929 mentioned this issue Nov 28, 2020

サーバーへデプロイ後、音楽を再生しようとすると Status code: 429 エラーが発生する dashimaki929/discord-jukebox#2

Closed

Error: input stream: Status code: 429 #444

Error: input stream: Status code: 429 #444

Comments

Harmiox commented Jul 9, 2019 • edited

encloinc commented Jul 10, 2019

encloinc commented Jul 10, 2019

fent commented Jul 10, 2019

encloinc commented Jul 10, 2019 • edited

encloinc commented Jul 10, 2019 • edited

fent commented Jul 10, 2019

encloinc commented Jul 10, 2019 • edited

HcgRandon commented Jul 10, 2019

encloinc commented Jul 10, 2019

edunad commented Jul 10, 2019

encloinc commented Jul 10, 2019

mh4ck commented Jul 10, 2019

HcgRandon commented Jul 10, 2019

Perflyst commented Jul 10, 2019

odedhb commented Jul 10, 2019

mh4ck commented Jul 10, 2019 • edited

Harmiox commented Jul 11, 2019 • edited

encloinc commented Jul 11, 2019

mh4ck commented Jul 11, 2019

encloinc commented Jul 11, 2019 via email

mh4ck commented Jul 11, 2019

WaqasIbrahim commented Jul 11, 2019

mh4ck commented Jul 11, 2019

WaqasIbrahim commented Jul 11, 2019

omarroth commented Jul 11, 2019 • edited

mh4ck commented Jul 12, 2019

ShivamJoker commented Jul 12, 2019

mh4ck commented Jul 12, 2019 • edited

ShivamJoker commented Jul 12, 2019 • edited

katsumi143 commented Sep 2, 2019 • edited

Harmiox commented Sep 7, 2019

WaqasIbrahim commented Sep 13, 2019

fent commented Sep 14, 2019

WaqasIbrahim commented Sep 14, 2019

Atruo commented Oct 2, 2019

mahhov commented Aug 6, 2020 • edited

fent commented Aug 6, 2020

mahhov commented Aug 6, 2020

mh4ck commented Aug 6, 2020

mahhov commented Aug 6, 2020

mh4ck commented Aug 6, 2020 • edited

mahhov commented Aug 7, 2020

mahhov commented Aug 7, 2020

fent commented Aug 7, 2020

mahhov commented Aug 7, 2020 • edited

notadevps commented Aug 17, 2020

cchen028 commented Aug 18, 2020 • edited

mahhov commented Aug 19, 2020

notadevps commented Aug 19, 2020

mahhov commented Aug 19, 2020

fent commented Aug 19, 2020

cchen028 commented Aug 20, 2020

TehPigYT commented Sep 19, 2020 • edited

Harmiox commented Jul 9, 2019 •

edited

encloinc commented Jul 10, 2019 •

edited

encloinc commented Jul 10, 2019 •

edited

encloinc commented Jul 10, 2019 •

edited

mh4ck commented Jul 10, 2019 •

edited

Harmiox commented Jul 11, 2019 •

edited

omarroth commented Jul 11, 2019 •

edited

mh4ck commented Jul 12, 2019 •

edited

ShivamJoker commented Jul 12, 2019 •

edited

katsumi143 commented Sep 2, 2019 •

edited

mahhov commented Aug 6, 2020 •

edited

mh4ck commented Aug 6, 2020 •

edited

mahhov commented Aug 7, 2020 •

edited

cchen028 commented Aug 18, 2020 •

edited

TehPigYT commented Sep 19, 2020 •

edited