[Bug]: Is Audio Timestamp Calculation Correct?

[eliaspuurunen]

Operating System Version

Windows 10 Pro, Windows 11 Pro

OBS Version

30.0.2

NDI Tools Version

5.6

Describe the bug

I've been hunting down some audio/video drift issues over the last few days with OBS-NDI. Here's what I've been able to find. Note this is based on my understanding of reading through the code for both OBS and obs-ndi. It may be inaccurate.

• Each OBS source maintains a "last timestamp" and "next expected timestamp" for audio. If the source sends audio to OBS that is later than the expected timestamp, buffers get adjusted.
• If the source's audio buffer gets "too large," audio gets dropped instead of getting added to the buffer.
• OBS will resample audio it receives from a source to match the internal sample rate OBS is running at. So does NDI, depending on how you set up that source. In OBS-NDI, the source's sample rate is used and passed to OBS.
• The latest version of OBS-NDI allows use of the framesync APIs. This will do some resampling/stretching as well. Note this receive and decode code path is separate from the typical "receiver" path.
• Sources aren't compelled to send video/audio frames at regular intervals. OBS seems to not like it when it doesn't receive an audio frame for some time, then gets a frame. (You can try this with Screen Capture HX. Make no sound, then make some sounds, then make no sound.)
• Network vs. Source timing doesn't change much - just which frame property OBS looks at to determine a frame's timestamp. Each NDI frame has a timecode and a timestamp. Timecode is supposed to be nanoseconds since Unix epoch, but it's not guaranteed (My Mevos transmit time since last power on). Timestamp is time frame was submitted to the NDI SDK - which could be an issue because it's when a frame is passed to NDI by an NDI source for transmission, not when it was received by the receiver.

I believe there is a fundamental incompatibility between OBS' internal sync and the way OBS-NDI is computing timestamps. OBS-NDI passes timecode/timestamp direct to OBS, which I think could cause issues with sync.

Shouldn't the increment of the timestamp depend on the audio sample rate and the number of frames?

So say we get 1024 samples (that's what Screen Capture HX sends from my machine, same with my Mevos).

The increment should be:

NoSamples / SampleRate * 1000000

So:

1024 / 48000 * 1000000 = 21,333.3333

But, the timestamps or timecodes, they're not guaranteed to increment by the number of samples received. NDI sources can send audio "frames" whenever they want, and while best practice is outlined in the SDK, senders are not bound to follow it.

Here's what my Mevo camera gives for timestamps when receiving an audio frame.

Received 1024 x 2 @ 48000 audio samples - Timestamp 3667146240000 | Timecode 3667146240000
Received 1024 x 2 @ 48000 audio samples - Timestamp 3667146453333 | Timecode 3667146453333
Received 1024 x 2 @ 48000 audio samples - Timestamp 3667146666666 | Timecode 3667146666666

If we do have a connection hiccup, we could go quite a while between frames and timestamps.

If my interpretation is correct, the way that timestamps are being calculated for the OBS source are wrong no matter what sync method you choose, as the timestamp/timecode is not guaranteed to increase in nice discrete intervals - and OBS is checking for the next expected timestamp. If it's outside that range, I think we get into buffer issues (and if the buffer grows too large, OBS just dumps out the audio).

obs-ndi takes that timestamp and multiplies it by 100

366714645333300 - 366714624000000 = 21,333,300. Which is far different from the 21333 above. And again - the timestamp/timecode from NDI is not guaranteed to increment in discrete steps that match the amount of audio samples received.

Also - in this case, my Screen Capture HX instance and my Mevo Starts send 1024 samples of audio data. I have another app that sends 1920 samples per audio frame. Again, an NDI sender can send an arbitrary number of samples per audio "frame" it sends out.

From the documentation:

Timecode: can be generated by the NDI source or synthesized by the NDI source. If synthesized, it's the current system clock time since the Unix Epoch with 100 nanosecond precision. (Again - it can be whatever the sender wants it to be)

Timestamp: Generated by the NDI SDK, it's the time (in 100 ns intervals) when the frame was submitted to the SDK - and this is based on the sender's clock.

IMO, the plugin should be looking at the # of samples received by NDI, not the timestamp/timecode of the audio frame.

Could that be the cause of some sync problems?

I could be 100% wrong about this, but my hunch is OBS doesn't like the way timestamps/timecodes are being generated.

Steps to reproduce

No response

Expected behavior

No response

Screenshots

No response

Additional context

No response

[eliaspuurunen]

I forked the code and did my own timestamp calculation. What I'm doing is simple - take the NDI timestamp, and subtract the duration of the audio samples received from it - and use that as the OBS Source timestamp. Since NDI's timestamps are always guaranteed to be behind the local computer, that should shift the audio frame by the appropriate amount so OBS places it in the correct spot in its buffer.

In ndi_source_thread_process_audio3:

    long long duration_microseconds = (static_cast<long long>(ndi_audio_frame3->no_samples * 1000000)) / ndi_audio_frame3->sample_rate;
    long long duration_ns = duration_microseconds * 100;

    // obs_audio_frame->timestamp += (uint64_t)(
    //     duration_microseconds * 100
    // );

	// switch (config->sync_mode) {
	// case PROP_SYNC_NDI_TIMESTAMP:
	// 	obs_audio_frame->timestamp =
	// 		(uint64_t)(ndi_audio_frame3->timestamp * 100);
	// 	break;

	// case PROP_SYNC_NDI_SOURCE_TIMECODE:
	// 	obs_audio_frame->timestamp =
	// 		(uint64_t)(ndi_audio_frame3->timecode * 100);
	// 	break;
	// }

    obs_audio_frame->timestamp =
			(uint64_t)(ndi_audio_frame3->timestamp * 100);

    obs_audio_frame->timestamp -= duration_ns;

However, the moment I apply a filter, I still get some audio issues. Will need to investigate further.

[eliaspuurunen]

Update: calculating the timestamp increment/decrement using all integer operations results in rounding errors of a few hundred nanoseconds per bundle of samples. With 1024 samples/audio frame being received at 48000Hz, that works out to 15,984 ns per second, or 0.016 ms inaccuracy.

After switching to calculating the duration using double floating point, the timestamp seems to be OK. Doing some more testing.

Here's my new code.

    double exact_duration_ns = ((double)ndi_audio_frame3->no_samples / (double)ndi_audio_frame3->sample_rate)
        * 1000000000.0;

    uint64_t timestamp_increment = static_cast<uint64_t>(exact_duration_ns);

    uint64_t duration_ns = ((uint64_t)(ndi_audio_frame3->timestamp * 100)) - timestamp_increment;
    obs_audio_frame->timestamp = duration_ns;

Still seems to be some audio glitches, especially when transmitted over NDI Bridge. Still investigating.

[eliaspuurunen]

Further testing showed that this method would cause audio issues.

I tried another method suggested by the OBS folks - just use os_gettime_ns() as the timestamp for an audio frame.

obs_audio_frame->timestamp = os_gettime_ns();

So far, been running for 15 minutes without any glitches in the worst-case scenario (NDI Bridge). Will continue testing.

[eliaspuurunen]

Update: I'm doing something similar to what OBS does in the decklink driver.

    case PROP_SYNC_NDI_SYNTHETIC:
        obs_audio_frame->timestamp = os_gettime_ns();

        obs_audio_frame->timestamp -= util_mul_div64(
            ndi_audio_frame3->no_samples,
            1000000000ULL,
            ndi_audio_frame3->sample_rate);

I also added another sync mode called "Synthetic" which ignores the NDI timestamp/timecode. So far, the sync/stability of video seems to be better with my sources.

[ogmkp]

Hi, is it possible de get a build for linux of your modifications ?
I have the issue here, the audio drifts quickly and the only workarround I have is to disable/enable the sound checkbox in the plugin or disable/enable monitoring in the sound panel of OBS.
So the delay is not present on encoding.

[ogmkp]

For reference: #742 #695 #676 #628 #498 #485 #479 #379 #667

[ogmkp]

I don't know if it's related, but I've found that activating framesync causes a constant delay in the sound on the monitor, which doesn't happen when it's deactivated.

[datalooper]

Try this as a temp fix:
https://youtu.be/I-id0-LFhMY