DTLS handshake failed with error illegal_parameter(47)

[jimm98y]

After upgrading the library to the latest preview 6.1.0-pre, the DTLS handshake is now failing and WebRTC no longer connects. Downgrading back to the 6.0.12 fixes the issue. It looks like the error is coming from BouncyCastle and I found this was recently updated to the latest nuget.

It looks like inside BouncyCastle it fails due to a mismatch of certificate types: RSA vs ECDSA. So it might be a duplicate of #953

I am running on Windows 11 ARM64, VS 2022 17.8.3, NET8.

sipsorcery[0]
RTCPeerConnection DTLS handshake failed with error illegal_parameter(47).
warn: sipsorcery[0]
DTLS server raised unexpected alert: fatal(2), illegal_parameter(47).
warn: sipsorcery[0]
DTLS handshake as server failed. illegal_parameter(47)
Org.BouncyCastle.Tls.TlsFatalAlert: illegal_parameter(47)
at Org.BouncyCastle.Tls.TlsUtilities.CheckClientCertificateType(CertificateRequest certificateRequest, Int16 clientCertificateType, Int16 alertDescription)
at Org.BouncyCastle.Tls.TlsUtilities.VerifyCertificateVerifyClient(TlsServerContext serverContext, CertificateRequest certificateRequest, DigitallySigned certificateVerify, TlsHandshakeHash handshakeHash)
at Org.BouncyCastle.Tls.DtlsServerProtocol.ProcessCertificateVerify(ServerHandshakeState state, Byte[] body, TlsHandshakeHash handshakeHash)
at Org.BouncyCastle.Tls.DtlsServerProtocol.ServerHandshake(ServerHandshakeState state, DtlsRecordLayer recordLayer, DtlsRequest request)
at Org.BouncyCastle.Tls.DtlsServerProtocol.Accept(TlsServer server, DatagramTransport transport, DtlsRequest request)
at Org.BouncyCastle.Tls.DtlsServerProtocol.Accept(TlsServer server, DatagramTransport transport)
at SIPSorcery.Net.DtlsSrtpTransport.DoHandshakeAsServer(String& handshakeError)

[lukedukeus]

+1, just ran into this same thing as well

[sipsorcery]

BouncyCastle update reverted.

[lukedukeus]

Just tested with 6.1.1, its back to working perfectly!

[jimm98y]

Yes, it works perfectly fine now, thank you very much!

In the meantime, I was looking into the code trying to fix it and I wonder whether there is some reason for not supporting EC and trying to force RSA.

Anyway, it looks like it was caused by BouncyCastle changes in the certificate verification years ago (notice DtlsServerProtocol -> ProcessCertificateVerify method and compare it with the one in Portable.BouncyCastle@1.9.0):
bcgit/bc-csharp@68c795f

Maybe also this one, I'm not sure without debugging it (TlsUtils -> CheckClientCertificateType did not exist before this changeset):
bcgit/bc-csharp@54bd89a

I have verified that the issue is the same even in 2.0.0.

It's possible to workaround the error by extending DtlsServerProtocol and overriding ProcessCertificateVerify method with an empty body:

public class PatchedDtlsServerProtocol : DtlsServerProtocol
{
    protected override void ProcessCertificateVerify(ServerHandshakeState state, byte[] body, TlsHandshakeHash handshakeHash)
    { }
}

Then in DtlsSrtpTransport.cs all we have to do is change:

DtlsServerProtocol serverProtocol = new DtlsServerProtocol();

To:

DtlsServerProtocol serverProtocol = new PatchedDtlsServerProtocol();

After this change it starts working with BouncyCastle 2.2.1.

Unfortunately, this bypasses all the certificate checks in DTLS and because all fields of the ServerHandshakeState are internal, we'd have to use reflection to do our own checks which could break at any time. At this point I am not sure if this is an issue in how sipsorcery is implementing DTLS, or with the new version of BouncyCastle.

[lostmsu]

I worked around this issue with BouncyCastle 2.2.1 by disregarding RFC 8244 5.1 (at least on SIPSorcery side) and simply only using ECC if no other options are available. See the commit above.

With this commit I get Firefox being able to connect to my SIPSorcery again.

[moorecj]

This may be resolved with this pull request. #1105

I was informed that the mismatch would be an issue with newer versions of Bouncy Castle which seems to be what the issue is here.