[MTVA Archivum] Add new extractor #32589
Conversation
Add new extractor for MTVA Archivum site. Closes ytdl-org#21430
| 'id': 'M3-87720998249999359', | ||
| 'ext': 'mp4', | ||
| 'title': 'Kék egér', | ||
| 'description': 'Kék egér nem sokáig örülhet a napsütésnek, mert egy kölyökkutya azt hiszi, kutyáknak való játék ez a kék valami. A Kék egér tiltakozása ellenére csak akkor engedi el az egeret, amikor az elásott csontja helyét megtalálja a kutya. A menekülő egérke elbotlik egy fél perecben aminek nagyon megörül, de egy erőszakos galamb meghívatja magát a perecre. Némi ellenszolgáltatás, és egy jó tanács fejében az egészet felfalja.', |
There was a problem hiding this comment.
The test harness will suggest a variant of this long test value with a digest ('md5:...digest...') but this doesn't help to check when the site makes a minor change to the extracted value. I prefer something like:
| 'description': 'Kék egér nem sokáig örülhet a napsütésnek, mert egy kölyökkutya azt hiszi, kutyáknak való játék ez a kék valami. A Kék egér tiltakozása ellenére csak akkor engedi el az egeret, amikor az elásott csontja helyét megtalálja a kutya. A menekülő egérke elbotlik egy fél perecben aminek nagyon megörül, de egy erőszakos galamb meghívatja magát a perecre. Némi ellenszolgáltatás, és egy jó tanács fejében az egészet felfalja.', | |
| 'description': r're:Kék egér nem sokáig örülhet a napsütésnek, mert egy kölyökkutya azt hiszi, kutyáknak való játék ez a kék valami\. .+\.$', |
Then (especially, I guess, with Hungarian language skills!) a future maintainer can compare some non-matching extracted value to see if the site has just tweaked or even completely changed the value, or maybe a new extraction tactic would get something like the expected value.
| 'id': 'M3-59898941410999595', | ||
| 'ext': 'mp4', | ||
| 'title': 'Magyar retro', | ||
| 'description': 'MTVA Archívum', |
There was a problem hiding this comment.
If the site has this value for description, maybe omit it, since it's just repeating the site name?
| webpage = self._download_webpage(url, video_id) | ||
| json = self._download_json('https://archivum.mtva.hu/m3/stream?no_lb=1&target=' + video_id, video_id) | ||
| video_url = json['url'] | ||
| title = self._og_search_title(webpage) or self._html_search_regex( |
There was a problem hiding this comment.
You have to give the first call a default or it will raise and never try the second one.
| title = self._og_search_title(webpage) or self._html_search_regex( | |
| title = self._og_search_title(webpage, default=None) or self._html_search_regex( |
| description = self._og_search_description(webpage) or self._html_search_regex( | ||
| '<p class=\"active-full-description\">\n.+</p>', webpage, 'description') |
There was a problem hiding this comment.
description is optional and shouldn't raise. Also, the fallback call needs a group in the RE (how many times have I left it out!) and the RE can be improved, ...
| description = self._og_search_description(webpage) or self._html_search_regex( | |
| '<p class=\"active-full-description\">\n.+</p>', webpage, 'description') | |
| description = self._og_search_description(webpage) or self._html_search_regex( | |
| '''<p\s[^>]*\bclass=['"]active-full-description\b[^>]*>(.+)</p>''', webpage, 'description', | |
| default=None) |
... or perhaps use a completely different function.
| description = self._og_search_description(webpage) or self._html_search_regex( | |
| '<p class=\"active-full-description\">\n.+</p>', webpage, 'description') | |
| description = self._og_search_description(webpage) or get_element_by_class( | |
| 'active-full-description', webpage) |
| formats = self._extract_m3u8_formats( | ||
| video_url, video_id, 'mp4') | ||
| self._sort_formats(formats) |
There was a problem hiding this comment.
Consider moving this up to just after the title extraction. Then the optional extraction won't be done until we know that we have a valid media link(s).
Also, you may want to supply the parameters ext='mp4', m3u8_id='hls', which for whatever reason aren't the defaults. And try extracting with entry_protocol='m3u8_native', (now the yt-dlp default): if that works ([hlsnative] entries in the log) the site's M3U8 manifests can be downloaded without ffmpeg.
Please follow the guide below
xinto all the boxes [ ] relevant to your pull request (like that [x])Before submitting a pull request make sure you have:
In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:
What is the purpose of your pull request?
Description of your pull request and other information
Add new extractor for MTVA Archivum site.
Closes #21430