Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Missing Menu metadata #488

Open
2 tasks done
JJGO opened this issue Jun 20, 2023 · 3 comments
Open
2 tasks done

[Bug]: Missing Menu metadata #488

JJGO opened this issue Jun 20, 2023 · 3 comments
Labels
help wanted Extra attention is needed question Further information is requested

Comments

@JJGO
Copy link

JJGO commented Jun 20, 2023

I've read the documentation

Operating System

Debian 10 buster Kernel 4.19

Your Bug Report

Describe the bug

When downloading a video with chapters, chapter metadata is not included in the file

Steps To Reproduce

  1. Download a video with chapters
  2. Compare the mediainfo output with a file donwloaded using yt-dlp -f mp4 --embed-metadata

Expected behavior

The metadata should match that of yt-dlp

Relevant log output

Given the following video: https://www.youtube.com/watch?v=ixTlSrViaFc

Downloading it with latest TubeArchivist (with the Current metadata embed setting: True)

Mediainfo output:

General
Complete name                            : 20230301_ixTlSrViaFc_Random chiptune mix 55.mp4
Format                                   : MPEG-4
Format profile                           : Base Media
Codec ID                                 : isom (isom/iso2/mp41)
File size                                : 103 MiB
Duration                                 : 49 min 29 s
Overall bit rate mode                    : Variable
Overall bit rate                         : 291 kb/s
Movie name                               : Random chiptune mix 55
Performer                                : Krelez
Description                              : Tracklist: / 01 00:00 - Deathro - E.S.S.O. (ft. RRThiel) / 02 04:16 - Tony Thai - Ghost Town Funk / 03 07:45 - nu11 - The Tower Of Dreams / 04 13:02 - Karl Brueggemann - Moon Factory Zone / 05 15:34 - Rei8bit - Utilize The Individuality / 06 18:57 - scythe  - いつか笑い話 / 07 22:15 - tom_atom - Run To Avoid Being Late / 08 25:42 - Wool Pool -  Souvenir Selection / 09 28:19 - EX-LYD - Sunken City / 10 32:26 - Maak - No Rain No Rainbow / 11 36:47 - iDoge - Red Sun / 12 40:28 - SymphonicGames - Fire Run / 13 43:14 - kgb525 - Facing Armored Windsor / 14 45:44 - FearOfDark - Comfy Cutie Hangout Zone /  / Artists: / 01 https://soundcloud.com/8bitdeathro/esso / 02 https://soundcloud.com/midiman10/ghost-town-funk-sample-pack-contest-ix-halloween-edition / 03 https://soundcloud.com/nu11_ft/the-tower-of-dreams / 04 https://soundcloud.com/karl-brueggemann/moon-factory-zone / 05 https://soundcloud.com/rei-kaj/utilize-the-individuality-vrc6 / 06 https://soundcloud.com/scythe1005/itsuka / 07 https://soundcloud.com/tom_atom/run-to-avoid-being-late / 08 https://m.soundcloud.com/gubera/souvenir-selection / 09 https://soundcloud.com/ex-lyd/sunken-city / 10 https://soundcloud.com/maakmusic/no-rain-no-rainbow-it / 11 https://soundcloud.com/idoge-451382705/red-sun-box / 12 https://symphonicgames.bandcamp.com/album/cosmic-wave-radio-fm-ep / 13 https://battleofthebits.bandcamp.com/track/facing-armored-windsor / 14 https://soundcloud.com/fod-steve/comfy-cutie-hangout-zone
Recorded date                            : 20230301
Writing application                      : Lavf60.5.100
Comment                                  : https://www.youtube.com/watch?v=ixTlSrViaFc
LongDescription                          : Tracklist: / 01 00:00 - Deathro - E.S.S.O. (ft. RRThiel) / 02 04:16 - Tony Thai - Ghost Town Funk / 03 07:45 - nu11 - The Tower Of Dreams / 04 13:02 - Karl Brueggemann - Moon Factory Zone / 05 15:34 - Rei8bit - Utilize The Individuality / 06 18:57 - scythe  - いつか笑い話 / 07 22:15 - tom_atom - Run To Avoid Being Late / 08 25:42 - Wool Pool -  Souvenir Selection / 09 28:19 - EX-LYD - Sunken City / 10 32:26 - Maak - No Rain No Rainbow / 11 36:47 - iDoge - Red Sun / 12 40:28 - SymphonicGames - Fire Run / 13 43:14 - kgb525 - Facing Armored Windsor / 14 45:44 - FearOfDark - Comfy Cutie Hangout Zone /  / Artists: / 01 https://soundcloud.com/8bitdeathro/esso / 02 https://soundcloud.com/midiman10/ghost-town-funk-sample-pack-contest-ix-halloween-edition / 03 https://soundcloud.com/nu11_ft/the-tower-of-dreams / 04 https://soundcloud.com/karl-brueggemann/moon-factory-zone / 05 https://soundcloud.com/rei-kaj/utilize-the-individuality-vrc6 / 06 https://soundcloud.com/scythe1005/itsuka / 07 https://soundcloud.com/tom_atom/run-to-avoid-being-late / 08 https://m.soundcloud.com/gubera/souvenir-selection / 09 https://soundcloud.com/ex-lyd/sunken-city / 10 https://soundcloud.com/maakmusic/no-rain-no-rainbow-it / 11 https://soundcloud.com/idoge-451382705/red-sun-box / 12 https://symphonicgames.bandcamp.com/album/cosmic-wave-radio-fm-ep / 13 https://battleofthebits.bandcamp.com/track/facing-armored-windsor / 14 https://soundcloud.com/fod-steve/comfy-cutie-hangout-zone

Video
ID                                       : 1
Format                                   : vp09
Codec ID                                 : vp09
Duration                                 : 49 min 29 s
Bit rate                                 : 169 kb/s
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Variable
Frame rate                               : 6.000 FPS
Minimum frame rate                       : 5.988 FPS
Maximum frame rate                       : 6.024 FPS
Bits/(Pixel*Frame)                       : 0.014
Stream size                              : 60.0 MiB (58%)
Menus                                    : 0

Audio
ID                                       : 2
Format                                   : Opus
Codec ID                                 : Opus
Duration                                 : 49 min 29 s
Duration_FirstFrame                      : 1 ms
Bit rate mode                            : Variable
Bit rate                                 : 118 kb/s
Channel(s)                               : 2 channels
Sampling rate                            : 48.0 kHz
Bit depth                                : 16 bits
Compression mode                         : Lossy
Stream size                              : 41.9 MiB (41%)
Default                                  : Yes
Alternate group                          : 1
Menus                                    : 0

Menu

whereas downloading with yt-dlp -f mp4 --embed-metadata URL gives

Mediainfo output:

General
Complete name                            : Random chiptune mix 55 [ixTlSrViaFc].mp4
Format                                   : MPEG-4
Format profile                           : Base Media
Codec ID                                 : isom (isom/iso2/avc1/mp41)
File size                                : 64.7 MiB
Duration                                 : 49 min 29 s
Overall bit rate                         : 183 kb/s
Movie name                               : Random chiptune mix 55
Performer                                : Krelez
Description                              : Tracklist: / 01 00:00 - Deathro - E.S.S.O. (ft. RRThiel) / 02 04:16 - Tony Thai - Ghost Town Funk / 03 07:45 - nu11 - The Tower Of Dreams / 04 13:02 - Karl Brueggemann - Moon Factory Zone / 05 15:34 - Rei8bit - Utilize The Individuality / 06 18:57 - scythe  - いつか笑い話 / 07 22:15 - tom_atom - Run To Avoid Being Late / 08 25:42 - Wool Pool -  Souvenir Selection / 09 28:19 - EX-LYD - Sunken City / 10 32:26 - Maak - No Rain No Rainbow / 11 36:47 - iDoge - Red Sun / 12 40:28 - SymphonicGames - Fire Run / 13 43:14 - kgb525 - Facing Armored Windsor / 14 45:44 - FearOfDark - Comfy Cutie Hangout Zone /  / Artists: / 01 https://soundcloud.com/8bitdeathro/esso / 02 https://soundcloud.com/midiman10/ghost-town-funk-sample-pack-contest-ix-halloween-edition / 03 https://soundcloud.com/nu11_ft/the-tower-of-dreams / 04 https://soundcloud.com/karl-brueggemann/moon-factory-zone / 05 https://soundcloud.com/rei-kaj/utilize-the-individuality-vrc6 / 06 https://soundcloud.com/scythe1005/itsuka / 07 https://soundcloud.com/tom_atom/run-to-avoid-being-late / 08 https://m.soundcloud.com/gubera/souvenir-selection / 09 https://soundcloud.com/ex-lyd/sunken-city / 10 https://soundcloud.com/maakmusic/no-rain-no-rainbow-it / 11 https://soundcloud.com/idoge-451382705/red-sun-box / 12 https://symphonicgames.bandcamp.com/album/cosmic-wave-radio-fm-ep / 13 https://battleofthebits.bandcamp.com/track/facing-armored-windsor / 14 https://soundcloud.com/fod-steve/comfy-cutie-hangout-zone
Recorded date                            : 20230301
Writing application                      : Lavf58.45.100
Comment                                  : https://www.youtube.com/watch?v=ixTlSrViaFc
LongDescription                          : Tracklist: / 01 00:00 - Deathro - E.S.S.O. (ft. RRThiel) / 02 04:16 - Tony Thai - Ghost Town Funk / 03 07:45 - nu11 - The Tower Of Dreams / 04 13:02 - Karl Brueggemann - Moon Factory Zone / 05 15:34 - Rei8bit - Utilize The Individuality / 06 18:57 - scythe  - いつか笑い話 / 07 22:15 - tom_atom - Run To Avoid Being Late / 08 25:42 - Wool Pool -  Souvenir Selection / 09 28:19 - EX-LYD - Sunken City / 10 32:26 - Maak - No Rain No Rainbow / 11 36:47 - iDoge - Red Sun / 12 40:28 - SymphonicGames - Fire Run / 13 43:14 - kgb525 - Facing Armored Windsor / 14 45:44 - FearOfDark - Comfy Cutie Hangout Zone /  / Artists: / 01 https://soundcloud.com/8bitdeathro/esso / 02 https://soundcloud.com/midiman10/ghost-town-funk-sample-pack-contest-ix-halloween-edition / 03 https://soundcloud.com/nu11_ft/the-tower-of-dreams / 04 https://soundcloud.com/karl-brueggemann/moon-factory-zone / 05 https://soundcloud.com/rei-kaj/utilize-the-individuality-vrc6 / 06 https://soundcloud.com/scythe1005/itsuka / 07 https://soundcloud.com/tom_atom/run-to-avoid-being-late / 08 https://m.soundcloud.com/gubera/souvenir-selection / 09 https://soundcloud.com/ex-lyd/sunken-city / 10 https://soundcloud.com/maakmusic/no-rain-no-rainbow-it / 11 https://soundcloud.com/idoge-451382705/red-sun-box / 12 https://symphonicgames.bandcamp.com/album/cosmic-wave-radio-fm-ep / 13 https://battleofthebits.bandcamp.com/track/facing-armored-windsor / 14 https://soundcloud.com/fod-steve/comfy-cutie-hangout-zone

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : Main@L3.1
Format settings                          : CABAC / 3 Ref Frames
Format settings, CABAC                   : Yes
Format settings, Reference frames        : 3 frames
Codec ID                                 : avc1
Codec ID/Info                            : Advanced Video Coding
Duration                                 : 49 min 29 s
Bit rate                                 : 52.4 kb/s
Width                                    : 1 280 pixels
Height                                   : 720 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 6.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.009
Stream size                              : 18.5 MiB (29%)
Title                                    : ISO Media file produced by Google Inc. Created on: 03/01/2023.
Writing library                          : x264 core 155 r2901 7d0ff22
Color range                              : Limited
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709
Menus                                    : 3
Codec configuration box                  : avcC

Audio
ID                                       : 2
Format                                   : AAC LC
Format/Info                              : Advanced Audio Codec Low Complexity
Codec ID                                 : mp4a-40-2
Duration                                 : 49 min 29 s
Bit rate mode                            : Constant
Bit rate                                 : 128 kb/s
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 44.1 kHz
Frame rate                               : 43.066 FPS (1024 SPF)
Compression mode                         : Lossy
Stream size                              : 45.3 MiB (70%)
Title                                    : ISO Media file produced by Google Inc. Created on: 03/01/2023.
Language                                 : Russian
Default                                  : Yes
Alternate group                          : 1
Menus                                    : 3

Menu #1
ID                                       : 3
Format                                   : Timed Text
Codec ID                                 : text
Duration                                 : 49 min 29 s
Language                                 : English
Menu For                                 : 1,2
00:00:00.000                             : 01.Deathro - E.S.S.O. (ft. RRThiel)
00:04:16.000                             : 02.Tony Thai - Ghost Town Funk
00:07:45.000                             : 03.nu11 - The Tower Of Dreams
00:13:02.000                             : 04.Karl Brueggemann - Moon Factory Zone
00:15:34.000                             : 05.Rei8bit - Utilize The Individuality
00:18:57.000                             : 06.scythe - いつか笑い話
00:22:15.000                             : 07.tom_atom - Run To Avoid Being Late
00:25:42.000                             : 08.Wool Pool - Souvenir Selection
00:28:19.000                             : 09.EX-LYD - Sunken City
00:32:26.000                             : 10.Maak - No Rain No Rainbow
00:36:47.000                             : 11.iDoge - Red Sun
00:40:28.000                             : 12.SymphonicGames - Fire Run
00:43:14.000                             : 13.kgb525 - Facing Armored Windsor
00:45:44.000                             : 14.FearOfDark - Comfy Cutie Hangout Zone

Menu #2
00:00:00.000                             : 01.Deathro - E.S.S.O. (ft. RRThiel)
00:04:16.000                             : 02.Tony Thai - Ghost Town Funk
00:07:45.000                             : 03.nu11 - The Tower Of Dreams
00:13:02.000                             : 04.Karl Brueggemann - Moon Factory Zone
00:15:34.000                             : 05.Rei8bit - Utilize The Individuality
00:18:57.000                             : 06.scythe - いつか笑い話
00:22:15.000                             : 07.tom_atom - Run To Avoid Being Late
00:25:42.000                             : 08.Wool Pool - Souvenir Selection
00:28:19.000                             : 09.EX-LYD - Sunken City
00:32:26.000                             : 10.Maak - No Rain No Rainbow
00:36:47.000                             : 11.iDoge - Red Sun
00:40:28.000                             : 12.SymphonicGames - Fire Run
00:43:14.000                             : 13.kgb525 - Facing Armored Windsor
00:45:44.000                             : 14.FearOfDark - Comfy Cutie Hangout Zone
@bbilly1
Copy link
Member

bbilly1 commented Jun 21, 2023

We don't necessarily reproduce all the cli behavior of yt-dlp. Does yt-dlp python API have an option for chapters? At quick glance I couldn't find anything related to chapters. The cli seems to handle that differently:

--embed-metadata                Embed metadata to the video file. Also
                                embeds chapters/infojson if present unless
                                --no-embed-chapters/--no-embed-info-json are
                                used (Alias: --add-metadata)

If you find a way to access that from the info json, even better, then we can index it and also add it to the player.

Please help with that.

@bbilly1 bbilly1 added help wanted Extra attention is needed question Further information is requested labels Jun 21, 2023
@JJGO
Copy link
Author

JJGO commented Jun 22, 2023

Happy to help with this. It seems like the chapters are indeed included as part of the info json.

E.g. the command yt-dlp --dump-json 'https://www.youtube.com/watch?v=ixTlSrViaFc' | jq '.chapters' produces the following output

[
  {
    "start_time": 0,
    "title": "01.Deathro - E.S.S.O. (ft. RRThiel)",
    "end_time": 256
  },
  {
    "start_time": 256,
    "title": "02.Tony Thai - Ghost Town Funk",
    "end_time": 465
  },
  {
    "start_time": 465,
    "title": "03.nu11 - The Tower Of Dreams",
    "end_time": 782
  },
  {
    "start_time": 782,
    "title": "04.Karl Brueggemann - Moon Factory Zone",
    "end_time": 934
  },
  {
    "start_time": 934,
    "title": "05.Rei8bit - Utilize The Individuality",
    "end_time": 1137
  },
  {
    "start_time": 1137,
    "title": "06.scythe - いつか笑い話",
    "end_time": 1335
  },
  {
    "start_time": 1335,
    "title": "07.tom_atom - Run To Avoid Being Late",
    "end_time": 1542
  },
  {
    "start_time": 1542,
    "title": "08.Wool Pool - Souvenir Selection",
    "end_time": 1699
  },
  {
    "start_time": 1699,
    "title": "09.EX-LYD - Sunken City",
    "end_time": 1946
  },
  {
    "start_time": 1946,
    "title": "10.Maak - No Rain No Rainbow",
    "end_time": 2207
  },
  {
    "start_time": 2207,
    "title": "11.iDoge - Red Sun",
    "end_time": 2428
  },
  {
    "start_time": 2428,
    "title": "12.SymphonicGames - Fire Run",
    "end_time": 2594
  },
  {
    "start_time": 2594,
    "title": "13.kgb525 - Facing Armored Windsor",
    "end_time": 2744
  },
  {
    "start_time": 2744,
    "title": "14.FearOfDark - Comfy Cutie Hangout Zone",
    "end_time": 2969
  }
]

Additionally, I in order to reproduce the --embed-metadata behavior using the Python API, here's a minimal working example

import yt_dlp


def download_video(url):
    ydl_opts = {
        "format": "mp4",
        "postprocessors": [{
            "add_chapters": True,
            "add_infojson": "if_exists",
            "add_metadata": True,
            "key": "FFmpegMetadata",
        }]
    }

    with yt_dlp.YoutubeDL(ydl_opts) as ydl:
        ydl.download([url])


video_url = "https://www.youtube.com/watch?v=ixTlSrViaFc"
download_video(video_url)

Running mediainfo on the resulting file gives the same output as the CLI with --embed-metadata

@lamusmaser
Copy link
Collaborator

Looks like we have most of that already included here: https://github.com/tubearchivist/tubearchivist/blob/f848e73251d59de75671aeb97e08f064601a6042/tubearchivist/home/src/download/yt_dlp_handler.py#L293C10-L300

If you remove the add_infojson, do the chapters still generate appropriately? I would expect it to do so, but I want to confirm that it isn't some weird dependency issue that we just hadn't seen.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Extra attention is needed question Further information is requested
Projects
None yet
Development

No branches or pull requests

3 participants