stream.dash: Improve dynamic manifest reload logic

[michaelarnauts]

Checklist

• This is a plugin issue and not a different kind of issue
• I have read the contribution guidelines
• I have checked the list of open and recently closed plugin issues
• I have checked the commit log of the master branch

Streamlink version

6.2.1

Description

This is a follow-up of #5608 and the remark in #5610 (comment)

I'm seeing freezes in my stream because streamlink isn't downloading segments in due time. I'm going to quote @bastimeyer here, since he explains it better.

Yes, the manifest reload logic isn't ideal. It's basically still the same as the initial implementation from 2018 and way worse than the one of the HLS implementation. It requires a rewrite, so that the reload time can be calculated better.

Your manifest doesn't set a minimumUpdatePeriod, so the manifest reload time in the DASHStreamWorker falls back to the period duration and then back to a static value of 5 seconds. On top of that, if no new segments got queued, the reload wait time increases with a back-off factor of 1.3, which doesn't make any sense. It should lower the time of the next reload instead and it should also take the last segment length into consideration. It also reloads the manifest for each substream, meaning for both the video and audio streams, which is pretty bad.

Example manifest, although I think it applies on all dynamic MPEG DASH Manifests.

<?xml version="1.0" encoding="utf-8"?>
<MPD
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xmlns="urn:mpeg:dash:schema:mpd:2011"
 xmlns:ns1="http://www.w3.org/2001/XMLSchema-instance"
 xmlns:cenc="urn:mpeg:cenc:2013"
 xsi:schemaLocation="urn:mpeg:dash:schema:mpd:2011 DASH-MPD.xsd"
 profiles="urn:mpeg:dash:profile:isoff-live:2011,http://dashif.org/guidelines/dash-if-simple"
 maxSegmentDuration="PT5S" minBufferTime="PT5S" type="dynamic" publishTime="2023-10-12T13:18:26Z" timeShiftBufferDepth="PT1H0S"  availabilityStartTime="1970-01-01T00:00:00Z" minimumUpdatePeriod="PT0S">
 <Period id="p0" start="PT0S">
  <AdaptationSet mimeType="video/mp4" segmentAlignment="true" startWithSAP="1">
   <SegmentTemplate initialization="sc-gaFECw/$RepresentationID$_init.mp4" media="sc-gaFECw/$RepresentationID$_t$Time$.m4s" timescale="90000">
    <SegmentTimeline>
<S t="152740179303376" d="360000" r="899"/>
    </SegmentTimeline>
   </SegmentTemplate>
   <Representation id="V0" bandwidth="1255000" codecs="avc1.640028" frameRate="25" width="1920" height="1080" sar="1:1"/>
   <Representation id="V1" bandwidth="535000" codecs="avc1.64001f" frameRate="25" width="1280" height="720" sar="1:1"/>
   <Representation id="V2" bandwidth="353000" codecs="avc1.4d401f" frameRate="25" width="960" height="540" sar="1:1"/>
   <Representation id="V3" bandwidth="213000" codecs="avc1.4d401e" frameRate="25" width="768" height="432" sar="1:1"/>
   <Representation id="V4" bandwidth="157000" codecs="avc1.4d401e" frameRate="25" width="640" height="360" sar="1:1"/>
  </AdaptationSet>
  <AdaptationSet mimeType="audio/mp4" lang="nld" segmentAlignment="true" startWithSAP="1">
   <Role schemeIdUri="urn:mpeg:dash:role:2011" value="main"/>
   <SegmentTemplate initialization="sc-gaFECw/$RepresentationID$_init.mp4" media="sc-gaFECw/$RepresentationID$_t$Time$.m4s" timescale="90000">
    <SegmentTimeline>
<S t="152740179304696" d="359040"/>
<S d="360960"/>
<S d="359040"/>
<S d="360960"/>
<S d="359040"/>
<S d="360960"/>
<!-- cut -->
<S d="360960"/>
<S d="359040"/>
<S d="360960"/>
    </SegmentTimeline>
   </SegmentTemplate>
   <Representation id="A0" bandwidth="128000" codecs="mp4a.40.2" audioSamplingRate="48000"><AudioChannelConfiguration schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011" value="2"/></Representation>
  </AdaptationSet>
 </Period>
</MPD>

Debug log

[cli][debug] OS:         Linux-5.15.123.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
[cli][debug] Python:     3.10.12
[cli][debug] OpenSSL:    OpenSSL 3.0.2 15 Mar 2022
[cli][debug] Streamlink: 6.2.1+23.g0490c4ef.dirty
[cli][debug] Dependencies:
[cli][debug]  certifi: 2023.7.22
[cli][debug]  isodate: 0.6.1
[cli][debug]  lxml: 4.9.3
[cli][debug]  pycountry: 22.3.5
[cli][debug]  pycryptodome: 3.19.0
[cli][debug]  PySocks: 1.7.1
[cli][debug]  requests: 2.31.0
[cli][debug]  trio: 0.22.2
[cli][debug]  trio-websocket: 0.11.1
[cli][debug]  typing-extensions: 4.8.0
[cli][debug]  urllib3: 2.0.6
[cli][debug]  websocket-client: 1.6.4
[cli][debug] Arguments:
[cli][debug]  url=https://xxx/dash.mpd
[cli][debug]  stream=['best']
[cli][debug]  --loglevel=debug
[cli][debug]  --output=dump.ts
[cli][debug]  --ffmpeg-fout=mpegts
[cli][info] Found matching plugin dash for URL https://xxx/dash.mpd
[plugins.dash][debug] URL=https://xxx/dash.mpd; params={}
[utils.l10n][debug] Language code: nl_BE
[stream.dash][debug] Available languages for DASH audio streams: nld (using: nld)
[cli][info] Available streams: 360p (worst), 432p, 540p, 720p, 1080p (best)
[cli][info] Opening stream: 1080p (dash)
[cli][info] Writing output to
/home/michael/Development/streamlink/dump.ts
[cli][debug] Checking file output
[stream.dash][debug] Opening DASH reader for: ('p0', None, 'V0') - video/mp4
[stream.dash][debug] video/mp4 segment initialization: downloading (1970-01-01T00:00:00.000000Z / 2023-10-12T13:34:03.535015Z)
[stream.dash.manifest][debug] Generating segment timeline for dynamic playlist: ('p0', None, 'V0')
[stream.dash][debug] Opening DASH reader for: ('p0', None, 'A0') - audio/mp4
[stream.dash][debug] Reloading manifest ('p0', None, 'V0')
[stream.dash][debug] audio/mp4 segment initialization: downloading (1970-01-01T00:00:00.000000Z / 2023-10-12T13:34:03.538269Z)
[stream.dash.manifest][debug] Generating segment timeline for dynamic playlist: ('p0', None, 'A0')
[stream.dash][debug] Reloading manifest ('p0', None, 'A0')
[stream.dash][debug] video/mp4 segment 900: waiting 0.4s (2023-10-12T13:34:04.000000Z / 2023-10-12T13:34:03.555758Z)
[stream.dash][debug] video/mp4 segment initialization: completed
[stream.ffmpegmux][debug] ffmpeg version 6.0 Copyright (c) 2000-2023 the FFmpeg developers
[stream.ffmpegmux][debug]  built with gcc 11 (Ubuntu 11.3.0-1ubuntu1~22.04.1)
[stream.ffmpegmux][debug]  configuration: --disable-decoder=amrnb --disable-decoder=libopenjpeg --disable-gnutls --disable-liblensfun --disable-libopencv --disable-podpages --disable-sndio --disable-stripping --enable-avfilter --enable-chromaprint --enable-frei0r --enable-gcrypt --enable-gpl --enable-ladspa --enable-libaom --enable-libaribb24 --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libdc1394 --enable-libdrm --enable-libfdk-aac --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libglslang --enable-libgme --enable-libgsm --enable-libiec61883 --enable-libjack --enable-libkvazaar --enable-libmp3lame --enable-libmysofa --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librist --enable-librsvg --enable-librubberband --enable-libshine --enable-libsmbclient --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libsvtav1 --enable-libtesseract --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-nonfree --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-openssl --enable-postproc --enable-pthreads --enable-shared --enable-version3 --enable-vulkan --incdir=/usr/include/x86_64-linux-gnu --libdir=/usr/lib/x86_64-linux-gnu --prefix=/usr --toolchain=hardened --enable-vaapi --enable-libvpl --enable-libvmaf --enable-libdavs2 --enable-libxavs2 --enable-libilbc --enable-libjxl --disable-altivec --shlibdir=/usr/lib/x86_64-linux-gnu
[stream.ffmpegmux][debug]  libavutil      58.  2.100 / 58.  2.100
[stream.ffmpegmux][debug]  libavcodec     60.  3.100 / 60.  3.100
[stream.ffmpegmux][debug]  libavformat    60.  3.100 / 60.  3.100
[stream.ffmpegmux][debug]  libavdevice    60.  1.100 / 60.  1.100
[stream.ffmpegmux][debug]  libavfilter     9.  3.100 /  9.  3.100
[stream.ffmpegmux][debug]  libswscale      7.  1.100 /  7.  1.100
[stream.ffmpegmux][debug]  libswresample   4. 10.100 /  4. 10.100
[stream.ffmpegmux][debug]  libpostproc    57.  1.100 / 57.  1.100
[utils.named_pipe][info] Creating pipe streamlinkpipe-66036-1-7821
[utils.named_pipe][info] Creating pipe streamlinkpipe-66036-2-9899
[stream.ffmpegmux][debug] ffmpeg command: /usr/bin/ffmpeg -nostats -y -i /tmp/streamlinkpipe-66036-1-7821 -i /tmp/streamlinkpipe-66036-2-9899 -c:v copy -c:a copy -copyts -f mpegts pipe:1
[stream.ffmpegmux][debug] Starting copy to pipe: /tmp/streamlinkpipe-66036-1-7821
[stream.ffmpegmux][debug] Starting copy to pipe: /tmp/streamlinkpipe-66036-2-9899
[cli][debug] Pre-buffering 8192 bytes
[stream.dash][debug] audio/mp4 segment 900: waiting 0.1s (2023-10-12T13:34:04.000000Z / 2023-10-12T13:34:03.912906Z)
[stream.dash][debug] audio/mp4 segment initialization: completed
[stream.dash][debug] video/mp4 segment 900: downloading (2023-10-12T13:34:04.000000Z / 2023-10-12T13:34:04.000367Z)
[stream.dash][debug] audio/mp4 segment 900: downloading (2023-10-12T13:34:04.000000Z / 2023-10-12T13:34:04.001298Z)
[stream.dash][debug] audio/mp4 segment 900: completed
[stream.dash][debug] video/mp4 segment 900: completed
[cli][debug] Writing stream to output
[download] Written 1.42 MiB to dump.ts (4s @ 387.32 KiB/s)
[stream.dash.manifest][debug] Generating segment timeline for dynamic playlist: ('p0', None, 'V0')
[stream.dash][debug] Reloading manifest ('p0', None, 'V0')
[stream.dash.manifest][debug] Generating segment timeline for dynamic playlist: ('p0', None, 'A0')
[stream.dash][debug] Reloading manifest ('p0', None, 'A0')
[download] Written 1.42 MiB to dump.ts (11s @ 127.71 KiB/s)
[stream.dash.manifest][debug] Generating segment timeline for dynamic playlist: ('p0', None, 'V0')
[stream.dash][debug] Reloading manifest ('p0', None, 'V0')
[stream.dash][debug] video/mp4 segment 900: downloading (2023-10-12T13:34:10.000000Z / 2023-10-12T13:34:16.157370Z)
[stream.dash.manifest][debug] Generating segment timeline for dynamic playlist: ('p0', None, 'A0')
[stream.dash][debug] Reloading manifest ('p0', None, 'A0')
[stream.dash][debug] audio/mp4 segment 900: downloading (2023-10-12T13:34:10.000000Z / 2023-10-12T13:34:16.165648Z)
[stream.dash][debug] audio/mp4 segment 900: completed
[download] Written 1.46 MiB to dump.ts (11s @ 129.09 KiB/s)
[stream.dash][debug] video/mp4 segment 900: completed
[download] Written 2.95 MiB to dump.ts (16s @ 184.29 KiB/s)
[stream.dash.manifest][debug] Generating segment timeline for dynamic playlist: ('p0', None, 'V0')
[stream.dash][debug] Reloading manifest ('p0', None, 'V0')
[stream.dash][debug] video/mp4 segment 899: downloading (2023-10-12T13:34:12.000000Z / 2023-10-12T13:34:21.157445Z)
[stream.dash.manifest][debug] Generating segment timeline for dynamic playlist: ('p0', None, 'A0')
[stream.dash][debug] Reloading manifest ('p0', None, 'A0')
[stream.dash][debug] audio/mp4 segment 899: downloading (2023-10-12T13:34:11.989333Z / 2023-10-12T13:34:21.162231Z)
[download] Written 2.95 MiB to dump.ts (16s @ 181.52 KiB/s)
[stream.dash][debug] video/mp4 segment 900: downloading (2023-10-12T13:34:16.000000Z / 2023-10-12T13:34:21.402217Z)
[stream.dash][debug] video/mp4 segment 899: completed
[stream.dash][debug] audio/mp4 segment 900: downloading (2023-10-12T13:34:16.000000Z / 2023-10-12T13:34:21.408679Z)
[stream.dash][debug] audio/mp4 segment 899: completed
[stream.dash][debug] audio/mp4 segment 900: completed
[download] Written 4.20 MiB to dump.ts (17s @ 254.53 KiB/s)
[stream.dash][debug] video/mp4 segment 900: completed
[download] Written 5.84 MiB to dump.ts (18s @ 334.09 KiB/s)
^C
[stream.ffmpegmux][debug] Closing ffmpeg thread
[stream.segmented][debug] Closing worker thread
[stream.segmented][debug] Closing writer thread
[stream.ffmpegmux][debug] Pipe copy complete: /tmp/streamlinkpipe-66036-1-7821
[stream.segmented][debug] Closing worker thread
[stream.segmented][debug] Closing writer thread
[stream.ffmpegmux][debug] Pipe copy complete: /tmp/streamlinkpipe-66036-2-9899
[stream.ffmpegmux][debug] Closed all the substreams
[cli][info] Stream ended
Interrupted! Exiting...
[cli][info] Closing currently open stream...