Handle streams with corrupted pts or dts timestamps #1426
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 4 commits
September 7, 2023 13:41
…mps during discontinuities. - Add feature flag to calculate timestampOffset for each segment, regardless of its timeline. - Use audio/video start time info from Transmuxer instead of probeTs.
Codecov Report
@@ Coverage Diff @@
## main #1426 +/- ##
==========================================
+ Coverage 85.75% 85.81% +0.05%
==========================================
Files 42 42
Lines 10284 10298 +14
Branches 2374 2377 +3
==========================================
+ Hits 8819 8837 +18
+ Misses 1465 1461 -4
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
|
@dzianis-dashkevich I tested this PR in relation to #1371 and #1049 and it seems that it can also solve these two issues, allowing videojs/mux.js#430 and videojs/mux.js#437 to be closed. It would be great to have 1 or 2 unit tests. |
adrums86
reviewed
Sep 11, 2023
@amtins Regarding closing PR in mux.js... |
It seems like everything is passing now |
adrums86
approved these changes
Sep 11, 2023
wseymour15
approved these changes
Sep 15, 2023
dzianis-dashkevich
deleted the
handle-streams-with-corrupted-pts-or-dts-timestamps
branch
dzianis-dashkevich
added a commit
that referenced
this pull request
Dec 4, 2023
…rEachSegment and remove replaceSegmentsUntil (#1457) * Revert "fix: check for transmuxer for vtt-segment-loader (#1452)" This reverts commit b4dd748. * Revert "fix: fix several issues with calculate timestamp offset for each segment (#1451)" This reverts commit 3bbc6ef. * Revert "fix: replaceSegmentsUntil flag resetting too early (#1444)" This reverts commit af39ba5. * Revert "fix: prevent wrapping in resetMainLoaderReplaceSegments (#1439)" This reverts commit 719b7f4. * Revert "feat: Add feature flag to calculate timestampOffset for each segment to handle streams with corrupted pts or dts timestamps (#1426)" This reverts commit 2355ddc. * Revert "fix: fastQualityChange refactor (#1414)" This reverts commit 4590bdd. * cherry-pick: use transmuxer time info instead of probeTs * feat: sync controller media sequence strategy (#1458) * feat: add media sequence sync strategy * fix: fix current media sequence increment * chore: update logs * feat: use exact segment match in sync-controller * fix: fix race condition for a fast quality switch * chore: add additional logs for choose next request * feat: force timestamp after resync * chore: fix or skip tests * Update src/segment-loader.js Co-authored-by: Walter Seymour <walterseymour15@gmail.com> --------- Co-authored-by: Dzianis Dashkevich <ddashkevich@brightcove.com> Co-authored-by: Walter Seymour <walterseymour15@gmail.com> --------- Co-authored-by: Dzianis Dashkevich <ddashkevich@brightcove.com> Co-authored-by: Walter Seymour <walterseymour15@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Sometimes, we may encounter streams with corrupted PTS/DTS timestamps during discontinuities.
Let's review the following
tsanalyzedump based on these corrupted segments (This specific segment is the very first segment after discontinuity):As you can see, Audio packets have only PTS, and it is monotonically increasing from
0to180,480.Video packets have both PTS and DTS values, and DTS is not monotonically increasing:
It goes from
8,589,916,592to165,075.Let's take a look closer at video packets with timestamp info from this segment using
tsdump:1st video packet with PTS/DTS info:
PTS is
0x000000000, which is 0, and DTS is0x1FFFFB9B0, which is8589916592(33-bit)* Packet 2 ---- TS Header ---- PID: 258 (0x0102), header size: 96, sync: 0x47 Error: 0, unit start: 1, priority: 0 Scrambling: 0, continuity counter: 0 Adaptation field: yes (92 bytes), payload: yes (92 bytes) Discontinuity: 0, random access: 0, ES priority: 0 ---- PES Header ---- Stream id: 0xE0 (Video 0) PES packet length: 0 (unbounded) DTS: 0x1FFFFB9B0, PTS: 0x000000000 (DTS-95,443,517 ms) ---- Full TS Packet Content ---- 47 41 02 30 5B 00 FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 00 00 01 E0 00 00 84 C0 0A 31 00 01 00 01 1F FF FF 73 61 00 00 00 01 09 10 00 00 00 01 67 4D 40 0D EC A0 D0 FF C9 35 01 01 01 A5 00 00 03 00 01 00 00 03 00 1E 8F 14 29 96 00 00 00 01 68 EA EF 20 00 00 01 06 05 FF FF EF DC 45 E9 BD E6 D9 48 B7 96 2C D8 20 D9 23 EE EF 78 32 362nd video packet with PTS/DTS info:
PTS is
0x000001770, which is6000, and DTS is0x1FFFFD120, which is8589922592(increased by6000, still 33-bit)* Packet 22 ---- TS Header ---- PID: 258 (0x0102), header size: 4, sync: 0x47 Error: 0, unit start: 1, priority: 0 Scrambling: 0, continuity counter: 4 Adaptation field: no (0 bytes), payload: yes (184 bytes) ---- PES Header ---- Stream id: 0xE0 (Video 0) PES packet length: 0 (unbounded) DTS: 0x1FFFFD120, PTS: 0x000001770 (DTS-95,443,517 ms) ---- Full TS Packet Content ---- 47 41 02 14 00 00 01 E0 00 00 84 C0 0A 31 00 01 2E E1 1F FF FF A2 41 00 00 00 01 09 30 00 00 01 41 9A 21 6C 44 7F DD E6 67 C8 FB 94 B9 FF 3E 99 5C BF C3 55 69 8C 41 CD 0B 0B E9 FC B8 33 65 99 29 12 83 31 6B BC 2B AD F5 C5 C3 A4 81 00 3C EE 2A E9 B2 3B A4 C3 88 C2 46 18 85 8E 94 78 69 63 44 BC 24 98 47 15 A5 59 44 59 41 88 72 42 7C 61 26 E0 72 92 80 0A C2 49 EE B7 D5 8F 7C 70 E4 E2 5B BE 36 84 9A DC CE DF 27 15 86 4B 53 32 5E 1E BA AC 29 4C E6 36 A2 CC 2D E5 F9 E6 C5 32 1D 10 19 36 D6 93 2F C9 31 DC 4E 90 59 AA EA A6 5D 59 CC 63 93 0B 6F 6F 48 80 A3 80 8A 743rd video packet with PTS/DTS info:
PTS is
0x000002EE0, which is12000, and DTS is0x1FFFFE890, which is8589928592(increased by6000, still 33-bit)* Packet 29 ---- TS Header ---- PID: 258 (0x0102), header size: 4, sync: 0x47 Error: 0, unit start: 1, priority: 0 Scrambling: 0, continuity counter: 11 Adaptation field: no (0 bytes), payload: yes (184 bytes) ---- PES Header ---- Stream id: 0xE0 (Video 0) PES packet length: 0 (unbounded) DTS: 0x1FFFFE890, PTS: 0x000002EE0 (DTS-95,443,517 ms) ---- Full TS Packet Content ---- 47 41 02 1B 00 00 01 E0 00 00 84 C0 0A 31 00 01 5D C1 1F FF FF D1 21 00 00 00 01 09 30 00 00 01 41 9A 42 18 26 FF D4 C3 C8 03 28 91 BC 9E 08 B8 8D 78 81 A7 27 7C 46 D5 50 F2 06 AE DE DE 25 EB B9 AB 3C 4F F1 25 BE 1B 1D 9C CB AC FE 50 84 DD F5 98 09 2E E3 9C ED DB 56 05 E3 90 27 20 35 0D A6 13 64 87 75 D0 A0 7D 96 8A 88 EC 26 A2 8C 82 BE 59 FD B9 3A 21 BC 57 43 59 5E 05 ED DB 56 D8 68 E9 B2 1B 57 06 8D 9D B1 0F E8 E0 A6 44 6D 19 9F C5 52 DE 18 88 D6 58 C8 17 38 97 AE 48 90 FA EF 95 9A D4 23 D7 0D C0 BB 68 E9 2E 42 6D 0F 70 55 05 F8 18 9D 42 94 28 CA 15 E8 AA4th video packet with PTS/DTS info:
PTS is
0x000004650, which is18000, and DTS is0x000000000, which is0(it seems like timestamp rollover because according to ISO/IEC 13818-1, DTS/PTS value should be a 33-bit number, but if we take previous value8589928592+6000changes it will be8589934592, which is1000000000000000000000000000000000in binary and this is 34-bit value, so it is rolling over to0)* Packet 46 ---- TS Header ---- PID: 258 (0x0102), header size: 4, sync: 0x47 Error: 0, unit start: 1, priority: 0 Scrambling: 0, continuity counter: 11 Adaptation field: no (0 bytes), payload: yes (184 bytes) ---- PES Header ---- Stream id: 0xE0 (Video 0) PES packet length: 0 (unbounded) DTS: 0x000000000, PTS: 0x000004650 (DTS+200 ms) ---- Full TS Packet Content ---- 47 41 02 1B 00 00 01 E0 00 00 84 C0 0A 31 00 01 8C A1 11 00 01 00 01 00 00 00 01 09 30 00 00 01 41 9A 63 18 26 FF D8 4E 46 D7 92 0A A9 04 4B 39 9B 54 1B 16 1E 76 3C 78 32 81 EE 7B 0B 62 19 09 1F 37 91 E4 4C 2E 3B 19 9C 09 8E 8A 43 4E 4B F8 53 02 6F 86 DC 17 C1 DF B9 49 47 23 8D 65 25 04 6A D4 E6 80 73 CC 07 88 71 E2 F6 D5 BE D2 32 8B 65 02 1A 53 E4 9D 9B 36 5C BD 8E 28 84 76 9F 3B A8 B8 E5 78 EB FF 39 12 14 A4 DC 68 CC C6 E5 FA 72 72 3C D8 8C 60 CB C0 05 AE AC A9 77 A1 78 3F 81 8A 98 D4 5E 13 20 76 22 0B BF C3 F8 9A C8 0C AE C6 4E 7F 28 49 7F 55 C9 D0 1E A4Since mux.js relies heavily on DTS values, we have two problems here:
Problem 1:
Differences between audio and video packets' reported start time in the same segment from
probeTs:0vs8589916592Problem 2:
Wrong timestampOffset for next segment after corrupted one.
Since this is a discontinuity segment, we re-calculate
timestampOffsets based on the segment's timestamps and apply them to source buffers.The next segment will have the same timeline as the corrupted segment. So We do not re-calculate
timestampOffset, but since the timestamp value was rolled over, we should apply a different timestampOffset for proper time alignment.Specific Changes proposed
probeTsbecause Transmuxer aligns time between audio and video in the timestampRollover stream.timestampOffsetfor each segment, regardless of its timeline, so timestampOffset should always be valid.I added a feature flag to be safer, but overall, I consider this change risky, as it may break something.
Requirements Checklist