youtube channel page downloads broken #148

bani6809 · 2020-11-09T20:24:21Z

Checklist

I'm reporting a broken site support issue
I've verified that I'm running youtube-dlc version 2020.10.31
I've checked that all provided URLs are alive and playable in a browser
I've checked that all URLs and arguments with special characters are properly quoted or escaped
I've searched the bugtracker for similar bug reports including closed ones
I've read bugs section in FAQ

Verbose log

youtube-dlc.exe --ignore-config -i -o "%(uploader)s (%(uploader_id)s)/%(upload_date)s - %(title)s - (%(duration)ss) [%(resolution)s] [%(id)s].%(ext)s" --prefer-ffmpeg --merge-output-format mkv --write-sub --all-subs --convert-subs srt --add-metadata --write-description --write-annotations --write-all-thumbnails --write-info-json --cookies youtube-cookies.txt -v --proxy "" https://www.youtube.com/channel/UC7qPftDWPw9XuExpSgfkmJQ
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--ignore-config', '-i', '-o', '%(uploader)s (%(uploader_id)s)/%(upload_date)s - %(title)s - (%(duration)ss) [%(resolution)s] [%(id)s].%(ext)s', '--prefer-ffmpeg', '--merge-output-format', 'mkv', '--write-sub', '--all-subs', '--convert-subs', 'srt', '--add-metadata', '--write-description', '--write-annotations', '--write-all-thumbnails', '--write-info-json', '--cookies', 'youtube-cookies.txt', '-v', '--proxy', '', 'https://www.youtube.com/channel/UC7qPftDWPw9XuExpSgfkmJQ']
[debug] Loading archive file None
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252
[debug] youtube-dlc version 2020.11.07
[debug] Python version 3.9.0 (CPython) - Windows-10-10.0.19041-SP0
[debug] exe versions: ffmpeg git-2020-04-24-a501947, ffprobe git-2020-04-24-a501947
[debug] Proxy map: {}
[youtube:channel] UC7qPftDWPw9XuExpSgfkmJQ: Downloading channel page
[youtube:playlist] UU7qPftDWPw9XuExpSgfkmJQ: Downloading webpage
[download] Downloading playlist: UU7qPftDWPw9XuExpSgfkmJQ
[youtube:playlist] playlist UU7qPftDWPw9XuExpSgfkmJQ: Downloading 0 videos
[download] Finished downloading playlist: UU7qPftDWPw9XuExpSgfkmJQ

Description

youtube playlist downloads seem to have broken sometime within the last few hours. it was working earlier this morning, now all playlists result in "0 videos" from youtube-dlc. presumably youtube made a schema change which broke youtube-dlc.

Individual video downloads still work, for example this works perfectly:

youtube-dlc.exe --ignore-config -i -o "%(uploader)s (%(uploader_id)s)/%(upload_date)s - %(title)s - (%(duration)ss) [%(resolution)s] [%(id)s].%(ext)s" --prefer-ffmpeg --merge-output-format mkv --write-sub --all-subs --convert-subs srt --add-metadata --write-description --write-annotations --write-all-thumbnails --write-info-json --cookies youtube-cookies.txt -v --proxy "" https://www.youtube.com/watch?v=SXGAiTjFiMA

The text was updated successfully, but these errors were encountered:

someziggyman · 2020-11-09T21:25:33Z

@insaneracist @blackjack4494
I can confirm that this is broken for links like:
https://www.youtube.com/user/anjunadeep/videos
and "regular" playlists like this ID: PLOftnzGIKwJAfH33eTU2wDsBqwxOMRedV
Full link: https://www.youtube.com/watch?v=mhpfOQ6sf3s&list=PLOftnzGIKwJAfH33eTU2wDsBqwxOMRedV
or this:
https://www.youtube.com/playlist?list=RDCLAK5uy_m_h-nx7OCFaq9AlyXv78lG0AuloqW_NUA

Mix, search and user created playlists seem to be working just fine:
mix id to test: RDKR9wGi7gVLQ
search playlist to test: https://www.youtube.com/results?search_query=linkin+park+numb

bani6809 · 2020-11-09T21:59:14Z

seems widely confirmed: https://www.reddit.com/r/youtubedl/comments/jr56fs/yourubedl_sees_all_channel_links_as_empty/

ainsey11 · 2020-11-09T22:08:51Z

I too am seeing this issue, looks to be a global thing, same problem wherever I initiate the download from

[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'https://www.youtube.com/playlist?list=PLvgS71fU12Mbx-w18Chu_Sg9v6loipEFO', u'--verbose']
[debug] Loading archive file None
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dlc version 2020.11.07
[debug] Python version 2.7.17 (CPython) - Linux-4.19.104-microsoft-standard-x86_64-with-Ubuntu-18.04-bionic
[debug] exe versions: ffmpeg 3.4.8, ffprobe 3.4.8, phantomjs ., rtmpdump 2.4
[debug] Proxy map: {}
[youtube:playlist] PLvgS71fU12Mbx-w18Chu_Sg9v6loipEFO: Downloading webpage
[download] Downloading playlist: PLvgS71fU12Mbx-w18Chu_Sg9v6loipEFO
[youtube:playlist] playlist PLvgS71fU12Mbx-w18Chu_Sg9v6loipEFO: Downloading 0 videos
[download] Finished downloading playlist: PLvgS71fU12Mbx-w18Chu_Sg9v6loipEFO```

ainsey11 · 2020-11-09T22:38:13Z

It would appear that youtube has disabled the "disable_polymer" feature that the YT extractor requires,

When visiting a playlist with the flag set, I now get unknown param

I can see the extractor is doing `

        query['disable_polymer'] = 'true'

so that is my hunch

jbruchon · 2020-11-09T23:36:45Z

@ainsey11 I can second what you've found there. Seems you're on the right track.

etec-masterofsynapse · 2020-11-10T00:38:42Z

Sorry if its a dumb question: Why does the extractor need the "old theme / non-polymer"?
Is it not currently able to pull the video IDs from the new theme?

wlritchi · 2020-11-10T01:21:37Z

Sorry if its a dumb question: Why does the extractor need the "old theme / non-polymer"?
Is it not currently able to pull the video IDs from the new theme?

The extractor currently looks for HTML markup for each entry in a playlist. When the polymer frontend was introduced, YouTube changed from server-side rendering of HTML templates to client-side rendering. As a result, the HTML source received from the server no longer contains the markup the extractor looks for, and the only place that playlist items appear is in a JSON structure (and a fairly dense one at that). Now that the extractor can no longer opt out of polymer, it will need to be updated to pull data out of that structure.

ZenulAbidin · 2020-11-10T02:25:21Z

It seems to be failing at youtube_dlc/extractor/youtube.py:_extract_playlist

https://github.com/blackjack4494/yt-dlc/blob/master/youtube_dlc/extractor/youtube.py#L2890-L2892

Every match after these lines isn't working.

candrapersada · 2020-11-10T04:51:02Z

Individual video downloads still work, for example this works perfectly

how to use playlist track number from Individual video downloads? &index=2
Full link https://www.youtube.com/watch?v=QW7vi-LjzXI&list=PLOftnzGIKwJAfH33eTU2wDsBqwxOMRedV&index=2

jbruchon · 2020-11-10T05:40:44Z

@candrapersada That's fine as a hack, but for all of us who do archival, it's a really bad hack to put in place. Using that method, there is no way to scan the archive for all of the playlist items and avoid downloading the page for every single one of those items. I archive over 150 channels regularly and dropping down to this method would be a little bit of a disaster.

jbruchon · 2020-11-10T05:42:05Z

If no one has taken up the task of getting this fixed yet, I might give it a try tomorrow.

blackjack4494 · 2020-11-10T06:12:29Z

Check network tab in browser. They likely completely changed over to api usage returning json instead of html events

candrapersada · 2020-11-10T06:33:07Z

what I mean is how to want youtube-dlc to name the files like 1-{name}, 2-{name}, ... n-{name} in order to be able to watch them in the same sequence as original Youtube playlist. In other words I need my downloaded videos to be prefixed with numbers. How can I do that?

insaneracist · 2020-11-10T09:24:17Z

@candrapersada, check out the readme section called output template
if playlist downloading isn't broken, you can do
youtube-dlc -o "%(playlist_index)03d - %(title)s.%(ext)s" "https://www.youtube.com/playlist?list=..."

arter97 · 2020-11-10T12:11:52Z

Hm, the playlist's title is still missing from these fixes @ZenulAbidin @insaneracist

Similar to #109.

python3.9 ./__main__.py --flat-playlist --add-metadata --all-subs -J -F 'https://youtube.com/playlist?list=PLq-W-cmkohAPAcxZWxh_ysJWPhqDxkETs'

Doesn't return a valid title key.

insaneracist · 2020-11-10T12:42:01Z

@arter97 fixed

blackjack4494 · 2020-11-10T12:47:24Z

Expect a release with a patch for this coming today (anything between 1-10 hrs depending on when I got time). Python version will be fixed then as well as win xp support is back then.

ZenulAbidin · 2020-11-10T19:20:35Z

I was booked today, now that I have some free time I’ll see where in the JSON our variables have moved to and hopefully find a regex to match them.

blackjack4494 · 2020-11-10T23:32:59Z

That's unfortunate
actions/upload-release-asset#68

Release is delayed till this issue is fixed.

perry-mitchell · 2020-11-11T12:46:05Z

@blackjack4494 Seems that the linked issue is fixed, according to comments. Would you have a potential ETA on release? Thank you 🙏

candrapersada · 2020-11-11T17:16:26Z

why name folder broken %(playlist)s/ and %(playlist_title)s/?

harrypotter0 · 2020-11-12T02:16:30Z

Hi everyone. I am trying to use this command to download playlist but not able to do so since past 2-3 days. Till now I can't see any fix for this. If anyone can tell me how to go past this issue it would be great help.

youtube-dl -c -i -f 18  https://www.youtube.com/playlist?list=PLv_5Gzr5T6wz0G1Ff3SO97ElLq1G0POWt
[youtube:playlist] PLv_5Gzr5T6wz0G1Ff3SO97ElLq1G0POWt: Downloading webpage
[download] Downloading playlist: PLv_5Gzr5T6wz0G1Ff3SO97ElLq1G0POWt
[youtube:playlist] playlist PLv_5Gzr5T6wz0G1Ff3SO97ElLq1G0POWt: Downloading 0 videos
[download] Finished downloading playlist: PLv_5Gzr5T6wz0G1Ff3SO97ElLq1G0POWt

Fabian42 · 2020-11-12T07:58:16Z

@harrypotter0 Which system are you using? For Manjaro Linux, a package update was available when I checked 9 hours ago, since then it works again. Some other distributions might take longer. Debian for example sometimes takes months, so you might want to download it from Github and install it that way instead.

someziggyman · 2020-11-12T08:12:12Z

Hi everyone. I am trying to use this command to download playlist but not able to do so since past 2-3 days. Till now I can't see any fix for this. If anyone can tell me how to go past this issue it would be great help.
youtube-dl -c -i -f 18  https://www.youtube.com/playlist?list=PLv_5Gzr5T6wz0G1Ff3SO97ElLq1G0POWt
[youtube:playlist] PLv_5Gzr5T6wz0G1Ff3SO97ElLq1G0POWt: Downloading webpage
[download] Downloading playlist: PLv_5Gzr5T6wz0G1Ff3SO97ElLq1G0POWt
[youtube:playlist] playlist PLv_5Gzr5T6wz0G1Ff3SO97ElLq1G0POWt: Downloading 0 videos
[download] Finished downloading playlist: PLv_5Gzr5T6wz0G1Ff3SO97ElLq1G0POWt

This particular playlist is working on the latest release.
try also using --rm-cache-dir --no-cache-dir params in your command.

cgerrie · 2020-11-13T07:50:50Z

When I download my Youtube watch later playlist, it only downloads the first page. Any videos beyond the first 100 don't get downloaded. This happens using both youtube-dlc --cookies cookies.txt :ytwatchlater and youtube-dlc --cookies cookies.txt https://www.youtube.com/playlist?list=WL. I am using version 2020.11.11-3, and I've cleared my cache directory. I've tried messing with the --playlist-start and --playlist-end flags, but those don't seem to help either. Could this be because the watch later playlist has some unique polymer structure that got missed in the patch?

HDark15 · 2020-11-16T00:54:12Z

When I download my Youtube watch later playlist, it only downloads the first page. Any videos beyond the first 100 don't get downloaded. This happens using both youtube-dlc --cookies cookies.txt :ytwatchlater and youtube-dlc --cookies cookies.txt https://www.youtube.com/playlist?list=WL. I am using version 2020.11.11-3, and I've cleared my cache directory. I've tried messing with the --playlist-start and --playlist-end flags, but those don't seem to help either. Could this be because the watch later playlist has some unique polymer structure that got missed in the patch?

The same happen to me. When the version 2020.11.11-3 came out, it fixes it. Now it broke again

HDark15 · 2020-11-16T01:01:46Z

This particular playlist is working on the latest release.
try also using --rm-cache-dir --no-cache-dir params in your command.

Can confirm that this works but for specific playlist

theAkito · 2020-12-02T17:00:30Z

I updated to today's youtube-dl version through pip. Still, none of the auxiliary options like --rm-cache-dir --no-cache-dir seem to work. Tried to download several completely different channels' videos and yet, it always tells me, that there are only 0 videos to download, which is absolutely false.

What am I missing here? Is this truly fixed?

blackjack4494 · 2020-12-02T17:07:02Z

Try updating to master with git. Search for git in readme :)

someziggyman mentioned this issue Nov 9, 2020

Fix #93 YoutubePlaylistsIE #147

Merged

9 tasks

insaneracist mentioned this issue Nov 10, 2020

[youtube] fix: playlist #150

Draft

8 tasks

wlritchi mentioned this issue Nov 10, 2020

RFC: youtube: Polymer UI and JSON endpoints for playlists #151

Merged

9 tasks

pukkandan mentioned this issue Nov 10, 2020

[Question] Too Many Requests! #158

Open

3 tasks

blackjack4494 closed this as completed in #151 Nov 10, 2020

This was referenced Nov 16, 2020

Channels/Playlists not working axcore/tartube#177

Closed

Cannot check/download channels/playlists on YouTube axcore/tartube#192

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

youtube channel page downloads broken #148

youtube channel page downloads broken #148

bani6809 commented Nov 9, 2020

someziggyman commented Nov 9, 2020 •

edited

bani6809 commented Nov 9, 2020

ainsey11 commented Nov 9, 2020 •

edited

ainsey11 commented Nov 9, 2020

jbruchon commented Nov 9, 2020

etec-masterofsynapse commented Nov 10, 2020

wlritchi commented Nov 10, 2020

ZenulAbidin commented Nov 10, 2020

candrapersada commented Nov 10, 2020 •

edited

jbruchon commented Nov 10, 2020

jbruchon commented Nov 10, 2020

blackjack4494 commented Nov 10, 2020

candrapersada commented Nov 10, 2020

insaneracist commented Nov 10, 2020

arter97 commented Nov 10, 2020

insaneracist commented Nov 10, 2020

blackjack4494 commented Nov 10, 2020

ZenulAbidin commented Nov 10, 2020

blackjack4494 commented Nov 10, 2020

perry-mitchell commented Nov 11, 2020

candrapersada commented Nov 11, 2020 •

edited

harrypotter0 commented Nov 12, 2020

Fabian42 commented Nov 12, 2020

someziggyman commented Nov 12, 2020

cgerrie commented Nov 13, 2020

HDark15 commented Nov 16, 2020

HDark15 commented Nov 16, 2020 •

edited

theAkito commented Dec 2, 2020

blackjack4494 commented Dec 2, 2020

youtube channel page downloads broken #148

youtube channel page downloads broken #148

Comments

bani6809 commented Nov 9, 2020

Checklist

Verbose log

Description

someziggyman commented Nov 9, 2020 • edited

bani6809 commented Nov 9, 2020

ainsey11 commented Nov 9, 2020 • edited

ainsey11 commented Nov 9, 2020

jbruchon commented Nov 9, 2020

etec-masterofsynapse commented Nov 10, 2020

wlritchi commented Nov 10, 2020

ZenulAbidin commented Nov 10, 2020

candrapersada commented Nov 10, 2020 • edited

jbruchon commented Nov 10, 2020

jbruchon commented Nov 10, 2020

blackjack4494 commented Nov 10, 2020

candrapersada commented Nov 10, 2020

insaneracist commented Nov 10, 2020

arter97 commented Nov 10, 2020

insaneracist commented Nov 10, 2020

blackjack4494 commented Nov 10, 2020

ZenulAbidin commented Nov 10, 2020

blackjack4494 commented Nov 10, 2020

perry-mitchell commented Nov 11, 2020

candrapersada commented Nov 11, 2020 • edited

harrypotter0 commented Nov 12, 2020

Fabian42 commented Nov 12, 2020

someziggyman commented Nov 12, 2020

cgerrie commented Nov 13, 2020

HDark15 commented Nov 16, 2020

HDark15 commented Nov 16, 2020 • edited

theAkito commented Dec 2, 2020

blackjack4494 commented Dec 2, 2020

someziggyman commented Nov 9, 2020 •

edited

ainsey11 commented Nov 9, 2020 •

edited

candrapersada commented Nov 10, 2020 •

edited

candrapersada commented Nov 11, 2020 •

edited

HDark15 commented Nov 16, 2020 •

edited