All of this is 100% just a matter of a missing marker bit when A's ringback stream is switched for the actual media stream from B - it's super apparent when adding instant_ringback into the mix.

The call starts bridging, and generated ringback immediately starts being sent to A with a timestamp starting from 0.
After td amount of time, the B side then responds with early media, also with timestamps starting from 0.
These two streams then have a more or less constant timestamp delta of td between them.
When B answers, FreeSWITCH discards the ringback stream, which was currently at timestamp tA, and A starts receiving media from B, currently at timestamp tB.
These new packets, n (starting from 0), suddenly have a timestamp of tA - td + (n * samples_per_interval), i.e. they are td amount of time old from A's perspective.
Since A has no idea that this is a new stream (as no SSRC has changed either), all packets from time tA until tA + td, when tB + td ≥ tA, will just look like duplicates.

In summary, a marker bit must be inserted when switching streams. Otherwise the receiver has no idea what just happened.

Further investigations indicate that Chrome (version 120) does not flush the jitter buffer and reset the stream upon receiving a marker bit, and thus a change of SSRC seems to be necessary.

The associated comment to CHANGE_SSRC_ON_MARKER suggests that FreeSWITCH will change the SSRC by default in this scenario, but we can see here that CHANGE_SSRC_ON_MARKER needs to be enabled in order to produce this behaviour.

In the same condition as mentioned above, there's also a check for the flag SWITCH_RTP_FLAG_SECURE_SEND, which is what ultimately prevents the change in SSRC in this case when using Verto (WebRTC/SRTP). This prevention was added way way back, but I find it peculiar that the condition was inverted between these two commits:
f4715781dc60293f89027ddf5ae4cc41badb5979
2fac3a8e83be837ff26d1010feb2be50119a10e8
The latter of the two mentions a refactor of video passthrough, so I'm wondering if this could be an oversight.

Anyhow, I made the necessary changes to run some tests and it all worked exactly as I had hoped it would - changing the SSRC fixes the issue caused by the apparent jump backwards in timestamps. I will get a PR up for this shortly.

For posterity; the gap in packets mentioned at the start of this issue becomes a non-issue by changing the SSRC.

Can you apply the below patch to the latest master and show me JDK... log lines?

diff --git a/src/switch_rtp.c b/src/switch_rtp.c
index 1125e2f59b..6ded1276ba 100644
--- a/src/switch_rtp.c
+++ b/src/switch_rtp.c
@@ -1561,6 +1561,14 @@ static uint8_t get_next_write_ts(switch_rtp_t *rtp_session, uint32_t timestamp)
                /* Send marker bit if timestamp is lower/same as before (resetted/new timer) */
                if (abs((int32_t)(rtp_session->ts - rtp_session->last_write_ts)) > rtp_session->samples_per_interval
                        && !(rtp_session->rtp_bugs & RTP_BUG_NEVER_SEND_MARKER)) {
+
+                       switch_log_printf(SWITCH_CHANNEL_SESSION_LOG(rtp_session->session), SWITCH_LOG_INFO,
+                             "JDK Increment m=[%d] rtp_session:[%p] rtp_session->ssrc:[%d] rtp_session->ts:[%d] "
+                             "rtp_session->last_write_ts:[%d] rtp_session->samples_per_interval:[%d] "
+                             "abs((int32_t)(rtp_session->ts - rtp_session->last_write_ts)) = [%d]\n",
+                             m, (void *)rtp_session, rtp_session->ssrc, rtp_session->ts, rtp_session->last_write_ts,
+                             rtp_session->samples_per_interval,
+                             abs((int32_t)(rtp_session->ts - rtp_session->last_write_ts)));
                        m++;
                }
        }

@jakubkarolczyk Sorry for the delay, but here we go!

I got one such log, at the moment when ringback starts being generated for the caller:
2024-02-27 14:02:20.610384 91.43% [INFO] switch_rtp.c:1565 JDK Increment m=[0] rtp_session:[0x7fec74184918] rtp_session->ssrc:[-837666077] rtp_session->ts:[13440] rtp_session->last_write_ts:[0] rtp_session->samples_per_interval:[960] abs((int32_t)(rtp_session->ts - rtp_session->last_write_ts)) = [13440]

Thanks @olofssonanton . Interesting, in my tests I always hit it second time when the actual audio starts after the ringback (so first packet after the gap you described). So I don't see the marker bit problem, what you reported.
Can you show me your dialplan, if possible?

Interesting indeed! Can you share some lines of debug log context around your second hit @jakubkarolczyk?

Do you have rtp_rewrite_timestamps chan var set by the chance?

@jakubkarolczyk rtp_rewrite_timestamps is not used

The biggest issue I see with this is that the timestamp is going to a lower value (inherited from the other leg) but the SSRC doesn't change. RFC 3550 says that the timestamp MUST be monotonically increasing. Genesys PBXs drop those because they think the packets are too long ago to be relevant, although most applications don't seem so strict.

We can't rely on the marker bit here because RTP is sent over UDP and we can't guarantee that a single packet will reach the end. Change the SSRC, which even feels correct because the source of the media has indeed changed.