[PornHub] Rename the age verification cookie #31916

arobase-che · 2023-03-23T00:37:27Z

Before submitting a pull request make sure you have:

Searched the bugtracker for similar pull requests
Read adding new extractor tutorial
Read youtube-dl coding conventions and adjusted the code to meet them
Covered the code with tests (note that PRs without tests will be REJECTED)
Checked the code with flake8

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

I am the original author of this code and I am willing to release it under Unlicense
I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

Bug fix
Improvement
New extractor
New feature

It only rename the legal age verification cookie.

dirkf · 2023-03-25T01:31:40Z

Does this solve an actual issue? I assume so, so there should either be a new or modified test for it, or just post verbose logs showing before and after.

arobase-che · 2023-03-29T03:51:34Z

Sorry, I didn't see your comment.
Yes it does solve an issue. The age verification cookie is now site dependent.
I couldn't try on the premium version but I just fix it for an alternative website supported by this extractor.

Here is the verbose output for 2 videos, with and without this path:

1 - Without:

$ youtube-dl --verbose "https://www.thumbzilla.com/video/ph62ac7b29e5a39/a-girl-s-perspective-part-2-trailer"
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', 'https://www.thumbzilla.com/video/ph62ac7b29e5a39/a-girl-s-perspective-part-2-trailer']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.10.10 (CPython) - Linux-6.2.8-arch1-1-x86_64-with-glibc2.37
[debug] exe versions: ffmpeg 6.0, ffprobe 6.0, rtmpdump 2.4
[debug] Proxy map: {}
[PornHub] ph62ac7b29e5a39: Downloading pc webpage
[PornHub] ph62ac7b29e5a39: Downloading tv webpage
ERROR: Unable to extract encoded url; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/youtube_dl/YoutubeDL.py", line 819, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/lib/python3.10/site-packages/youtube_dl/YoutubeDL.py", line 840, in __extract_info
    ie_result = ie.extract(url)
  File "/usr/lib/python3.10/site-packages/youtube_dl/extractor/common.py", line 535, in extract
    ie_result = self._real_extract(url)
  File "/usr/lib/python3.10/site-packages/youtube_dl/extractor/pornhub.py", line 400, in _real_extract
    js_vars = extract_js_vars(
  File "/usr/lib/python3.10/site-packages/youtube_dl/extractor/pornhub.py", line 337, in extract_js_vars
    assignments = self._search_regex(
  File "/usr/lib/python3.10/site-packages/youtube_dl/extractor/common.py", line 1013, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
youtube_dl.utils.RegexNotFoundError: Unable to extract encoded url; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

1 - With :

$ py youtube_dl/__main__.py --verbose "https://www.thumbzilla.com/video/ph62ac7b29e5a39/a-girl-s-perspective-part-2-trailer"
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', 'https://www.thumbzilla.com/video/ph62ac7b29e5a39/a-girl-s-perspective-part-2-trailer']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Git HEAD: 0b786108b
[debug] Python version 3.10.10 (CPython) - Linux-6.2.8-arch1-1-x86_64-with-glibc2.37
[debug] exe versions: ffmpeg 6.0, ffprobe 6.0, rtmpdump 2.4
[debug] Proxy map: {}
[PornHub] ph62ac7b29e5a39: Downloading pc webpage
[PornHub] ph62ac7b29e5a39: Downloading m3u8 information
[PornHub] ph62ac7b29e5a39: Downloading m3u8 information
[PornHub] ph62ac7b29e5a39: Downloading m3u8 information
[PornHub] ph62ac7b29e5a39: Downloading m3u8 information
[PornHub] ph62ac7b29e5a39: Downloading JSON metadata
WARNING: unable to extract view count; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'https://ev-h.phncdn.com/hls/videos/202206/17/410097631/,1080P_4000K,720P_4000K,480P_2000K,240P_1000K,_410097631.mp4.urlset/index-f1-v1-a1.m3u8?validfrom=1680058002&validto=1680065202&ipa=86.202.209.61&hdl=-1&hash=7vVYn8Prm%2FHCIzP2CN1Q89mZa78%3D'
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 10
[download] Destination: A Girl's Perspective Part 2 - TRAILER-ph62ac7b29e5a39.mp4
[download] 100% of 5.90MiB in 00:01
[debug] ffmpeg command line: ffprobe -show_streams 'file:A Girl'"'"'s Perspective Part 2 - TRAILER-ph62ac7b29e5a39.mp4'
[ffmpeg] Fixing malformed AAC bitstream in "A Girl's Perspective Part 2 - TRAILER-ph62ac7b29e5a39.mp4"
[debug] ffmpeg command line: ffmpeg -y -loglevel repeat+info -i 'file:A Girl'"'"'s Perspective Part 2 - TRAILER-ph62ac7b29e5a39.mp4' -c copy -f mp4 -bsf:a aac_adtstoasc 'file:A Girl'"'"'s Perspective Part 2 - TRAILER-ph62ac7b29e5a39.temp.mp4'

2 - Without

$ youtube-dl --verbose "https://www.pornhub.com/view_video.php?viewkey=ph61a74c98caef3"
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', 'https://www.pornhub.com/view_video.php?viewkey=ph61a74c98caef3']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.10.10 (CPython) - Linux-6.2.8-arch1-1-x86_64-with-glibc2.37
[debug] exe versions: ffmpeg 6.0, ffprobe 6.0, rtmpdump 2.4
[debug] Proxy map: {}
[PornHub] ph61a74c98caef3: Downloading pc webpage
[PornHub] ph61a74c98caef3: Downloading tv webpage
ERROR: Unable to extract encoded url; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/youtube_dl/YoutubeDL.py", line 819, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/lib/python3.10/site-packages/youtube_dl/YoutubeDL.py", line 840, in __extract_info
    ie_result = ie.extract(url)
  File "/usr/lib/python3.10/site-packages/youtube_dl/extractor/common.py", line 535, in extract
    ie_result = self._real_extract(url)
  File "/usr/lib/python3.10/site-packages/youtube_dl/extractor/pornhub.py", line 400, in _real_extract
    js_vars = extract_js_vars(
  File "/usr/lib/python3.10/site-packages/youtube_dl/extractor/pornhub.py", line 337, in extract_js_vars
    assignments = self._search_regex(
  File "/usr/lib/python3.10/site-packages/youtube_dl/extractor/common.py", line 1013, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
youtube_dl.utils.RegexNotFoundError: Unable to extract encoded url; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

2 - With

$ py youtube_dl/__main__.py --verbose 'https://www.pornhub.com/view_video.php?viewkey=ph61a74c98caef3'
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', 'https://www.pornhub.com/view_video.php?viewkey=ph61a74c98caef3']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Git HEAD: 0b786108b
[debug] Python version 3.10.10 (CPython) - Linux-6.2.8-arch1-1-x86_64-with-glibc2.37
[debug] exe versions: ffmpeg 6.0, ffprobe 6.0, rtmpdump 2.4
[debug] Proxy map: {}
[PornHub] ph61a74c98caef3: Downloading pc webpage
[PornHub] ph61a74c98caef3: Downloading m3u8 information
[PornHub] ph61a74c98caef3: Downloading m3u8 information
[PornHub] ph61a74c98caef3: Downloading m3u8 information
[PornHub] ph61a74c98caef3: Downloading JSON metadata
WARNING: unable to extract view count; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'https://cv-h.phncdn.com/hls/videos/202112/01/398949561/,720P_4000K,480P_2000K,240P_1000K,_398949561.mp4.urlset/index-f1-v1-a1.m3u8?v098K-Ule78xPnJI1a2j0bOy7sWumUk5TbFxBwr-2UVDc28Y0C18q3F3WlFQmGElOSWLu_akFFaKcFkqb67w4T5fp6dhwhfG9WAac49nhA-asHT726wxzrmRh5sReuRL4Bm-jgkz8dx12qrzZBJnuRcsr3uIPubYF3qmI-2lM4wxG3zgJmvyC25WrvFBOZ6VLGt_gS8OcA'
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 126
[download] Destination: アニメ声　量産型にオモチャ責め-ph61a74c98caef3.mp4
[download] 100% of 109.46MiB in 00:18
[debug] ffmpeg command line: ffprobe -show_streams 'file:アニメ声　量産型にオモチャ責め-ph61a74c98caef3.mp4'
[ffmpeg] Fixing malformed AAC bitstream in "アニメ声　量産型にオモチャ責め-ph61a74c98caef3.mp4"
[debug] ffmpeg command line: ffmpeg -y -loglevel repeat+info -i 'file:アニメ声　量産型にオモチャ責め-ph61a74c98caef3.mp4' -c copy -f mp4 -bsf:a aac_adtstoasc 'file:アニメ声　量産型にオモチャ責め-ph61a74c98caef3.temp.mp4'

Tests are broken because of a view count problem. I didn't investigate since it's not related to that PR directly.

dirkf · 2023-03-29T13:35:54Z

Are you also seeing yt-dlp/yt-dlp#4299?

arobase-che · 2023-03-29T15:48:14Z

That's curious. I didn't check out yt-dlp as I assume that they will get this fix ported from here.

But I suspect a geo-location issues as it mentions French ISP. Since I'm also in France, I will try from another country.
I will keep you updated.

arobase-che · 2023-03-29T17:25:29Z

Ok. So it's effectively a geo-location dependent behavior.
My patch correctly fix the extractor for me but it may breaks someone else extractor.

Is there is a location behavior aware mechanism already implemented we can import to fix that thing ?

dirkf · 2023-03-29T22:12:26Z

What happens if you set both the original and the ...XX cookies?

arobase-che · 2023-03-30T02:13:32Z

It just works. ^^

dirkf · 2023-03-30T15:51:13Z

youtube_dl/extractor/pornhub.py

        self._set_cookie(host, 'age_verified', '1')

+        if 'thumbzilla.com' in host:


Let's try the version from yt-dlp/yt-dlp#4299 (comment).

I let my comment about why we set multiple cookies. ^^
Feel free to edit it.

lenuxfrance · 2023-04-02T12:38:37Z

Hello, how can I modify my file?
what is the access path on linux
thanks

archusXIV · 2023-04-03T13:31:57Z

@lenuxfrance
from arch linux, As root in /usr/lib/python3.10/site-packages/yt_dlp/extractor/pornhub.py
lines 272 273

arobase-che · 2023-04-05T13:44:25Z

Can it be merged? Or does it need some modification ?

youtube_dl/extractor/pornhub.py

dirkf · 2023-04-07T23:29:43Z

From the discussion in the yt-dlp thread, the 3 new code lines should be moved to the beginning of the _login() method, after the test for already being logged in. This should fix the equivalent problem for playlists too.

arobase-che · 2023-04-11T08:32:57Z

Good idea ! I will do it by the day.

arobase-che · 2023-04-11T23:31:12Z

I can confirm it work even for playlist and model page (from French IP).

arobase-che · 2023-04-17T02:31:47Z

TZ Cookie doesn't seems to be needed as stated in the yt-dlp patch.

Can we merge it ?

Co-authored-by: bashonly <bashonly@bashonly.com> Co-authored-by: Noah <nkempers@outlook.de>

arobase-che · 2023-04-26T03:04:43Z

I adjust the commit to the version of yt-dlp and added two co-authors since they contribute at least as much as me on this PR.

The modification is about a new location (Amsterdam ?) that use another cookie name.

It's merged on the yt-dlp side.

arobase-che · 2023-05-11T08:43:26Z

Any plan of merging this ?

arobase-che · 2023-05-25T12:56:56Z

Hello ?

If I can be of any help, dont hesitate.

arobase-che · 2023-05-31T01:56:21Z

One month old, no news after multiple comments.
I'm out.

Rename the age verification cookie

ea3c33c

Add support for thumbzilla

0b78610

Update age verified cookies name logic

6fb9b53

bashonly mentioned this pull request Mar 30, 2023

[PornHub] Unable to extract encoded url yt-dlp/yt-dlp#4299

Closed

9 tasks

dirkf reviewed Mar 30, 2023

View reviewed changes

Alway set every cookies

e3aa7f9

bashonly mentioned this pull request Apr 2, 2023

[extractor/pornhub][fix] fix changed ageverification cookie yt-dlp/yt-dlp#6685

Merged

9 tasks

dirkf reviewed Apr 6, 2023

View reviewed changes

youtube_dl/extractor/pornhub.py Outdated Show resolved Hide resolved

Update comment

9517c05

dirkf changed the title ~~Rename the age verification cookie~~ [PornHub] Rename the age verification cookie Apr 7, 2023

Use a function to set age cookie

59b81f5

TZ cookie doesn't seems to be needed

aefed08

[PATCH] Set accessPH cookie

60ac3a6

Co-authored-by: bashonly <bashonly@bashonly.com> Co-authored-by: Noah <nkempers@outlook.de>

arobase-che closed this May 31, 2023

dirkf mentioned this pull request Aug 23, 2023

Fix pornhub info extractor #32503

Open

11 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[PornHub] Rename the age verification cookie #31916

[PornHub] Rename the age verification cookie #31916

arobase-che commented Mar 23, 2023

dirkf commented Mar 25, 2023

arobase-che commented Mar 29, 2023

dirkf commented Mar 29, 2023

arobase-che commented Mar 29, 2023

arobase-che commented Mar 29, 2023

dirkf commented Mar 29, 2023

arobase-che commented Mar 30, 2023

dirkf Mar 30, 2023

arobase-che Apr 2, 2023

lenuxfrance commented Apr 2, 2023

archusXIV commented Apr 3, 2023 •

edited

arobase-che commented Apr 5, 2023

dirkf commented Apr 7, 2023

arobase-che commented Apr 11, 2023

arobase-che commented Apr 11, 2023

arobase-che commented Apr 17, 2023

arobase-che commented Apr 26, 2023

arobase-che commented May 11, 2023

arobase-che commented May 25, 2023

arobase-che commented May 31, 2023 •

edited

		self._set_cookie(host, 'age_verified', '1')

		if 'thumbzilla.com' in host:

[PornHub] Rename the age verification cookie #31916

[PornHub] Rename the age verification cookie #31916

Conversation

arobase-che commented Mar 23, 2023

Before submitting a pull request make sure you have:

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

What is the purpose of your pull request?

dirkf commented Mar 25, 2023

arobase-che commented Mar 29, 2023

dirkf commented Mar 29, 2023

arobase-che commented Mar 29, 2023

arobase-che commented Mar 29, 2023

dirkf commented Mar 29, 2023

arobase-che commented Mar 30, 2023

dirkf Mar 30, 2023

Choose a reason for hiding this comment

arobase-che Apr 2, 2023

Choose a reason for hiding this comment

lenuxfrance commented Apr 2, 2023

archusXIV commented Apr 3, 2023 • edited

arobase-che commented Apr 5, 2023

dirkf commented Apr 7, 2023

arobase-che commented Apr 11, 2023

arobase-che commented Apr 11, 2023

arobase-che commented Apr 17, 2023

arobase-che commented Apr 26, 2023

arobase-che commented May 11, 2023

arobase-che commented May 25, 2023

arobase-che commented May 31, 2023 • edited

archusXIV commented Apr 3, 2023 •

edited

arobase-che commented May 31, 2023 •

edited