DTLS 1.2 broken in version 1.77; handshake finished sends hello_request instead of change_cipher_spec and finished

[mondain]

There is a bug in DtlsRecordLayer.send(byte[] buf, int off, int len) link when called with a HandshakeType.finished does not send change_cipher_spec nor finished and instead only sends hello_request. This prevent an DTLS client implementations from connecting to a DTLS 1.2 servers.

In DTLSClientProtocol around line 326 is where the call initiates:

        securityParameters.localVerifyData = TlsUtils.calculateVerifyData(clientContext, handshake.getHandshakeHash(),
            false);
        handshake.sendMessage(HandshakeType.finished, securityParameters.getLocalVerifyData());

Also in addition to this issue in the same class, I believe there is a typo at line 101 where the commented out line is actually more correct (if works where while does not):

        // TODO Consider stricter HelloVerifyRequest protocol
//        if (serverMessage.getType() == HandshakeType.hello_verify_request)
        while (serverMessage.getType() == HandshakeType.hello_verify_request)

Reference https://datatracker.ietf.org/doc/html/rfc4346#section-7.4.9

[mondain]

Sequence example from RFC 5764

sequenceDiagram
    participant X as Client
    participant Y as Server
    X->>Y: ClientHello
    Y->>X: ServerHello
    Y-->>X: Certificate
    Y-->>X: ServerKeyExchange
    Y-->>X: CertificateRequest
    Y->>X: ServerHelloDone
    X-->>Y: Certificate
    X-->>Y: ClientKeyExchange
    X-->>Y: CertificateVerify
    X->>Y: ChangeCipherSpec
    X->>Y: Finished
    Y->>X: ChangeCipherSpec
    Y->>X: Finished

Actual sequence in bctls 177 / 1.77

sequenceDiagram
    participant X as Client
    participant Y as Server
    X->>Y: ClientHello
    Y->>X: ServerHello
    Y-->>X: Certificate
    Y-->>X: ServerKeyExchange
    Y-->>X: CertificateRequest
    Y->>X: ServerHelloDone
    X-->>Y: Certificate
    X-->>Y: ClientKeyExchange
    X-->>Y: CertificateVerify
    X->>Y: HelloRequest

[mondain]

Truncated log output:

From the client end after we've received the server_hello_done and have sent up-to certificate_verify:

Send handshake certificate_verify
output stream write:

16
fe fd
00 00
00 00 00 00 00 03
01 10
0f
0001040003000000000104040101001efe8b5fa5b69c7acd5c5695e861e815956d7fa9f8223c8fadfd5494c75937929322027a97d08fa7593f5c0b2affda30de8f88198666cc7822661908a0cdf337d0a6b8ee742f81c63a60b14a5b742113c3d6079f3edf2e7f3ae502690c53395f34ee7baddd0906b608bab15018cb5593acb968b8144dbf58c01a45d712396fd0c01b23f25eede0910ad0cc380edb83b1de4b7d299c282866f3770de2879c6810e172eebaeedf02a2d6f2e32e8323d9d2993f56798fd7fb4ce7d153bcb2eb7970ae1dd32843e84c6b6f20a88eb4ab663b46bb9cc05a95085d162b0069c8f07dd651dddb40d305e8731837e7056bf1a92bc5194d93bd11c07990beacdfcbbd939d

Sent handshake certificate_verify

Send handshake finished

org.bouncycastle.tls.DTLSRecordLayer - send - buf: 1400000c000400000000000cb998963f6464e0fc82e4e1f0
org.bouncycastle.tls.DTLSRecordLayer - send - inHandshake: true writeEpoch: org.bouncycastle.tls.DTLSEpoch@49ac46f9 retransmitEpoch: null
org.bouncycastle.tls.DTLSRecordLayer - send finished - handshakeType: 20
org.bouncycastle.tls.DTLSRecordLayer - send change cipher spec
org.bouncycastle.tls.DTLSRecordLayer - sendRecord - contentType: 20 buf: 01
org.bouncycastle.tls.DTLSRecordLayer - getMacSequenceNumber - epoch: 0 sequence_number: 4
org.bouncycastle.tls.DTLSRecordLayer - sendDatagram: 14fefd0000000000000004000101

Send change_cipher_spec
output stream write:

14
fe fd
00 00
00 00 00 00 00 04
00 01
01

org.bouncycastle.tls.DTLSRecordLayer - sendRecord - contentType: 22 buf: 1400000c000400000000000cb998963f6464e0fc82e4e1f0
org.bouncycastle.tls.DTLSRecordLayer - getMacSequenceNumber - epoch: 1 sequence_number: 0
org.bouncycastle.tls.DTLSRecordLayer - sendDatagram: 16fefd00010000000000000030000100000000000037c2d8db838d0ec23b49682f7807834921ba1e712740f97a32974c5edf7a275393fe2381f405439c

Send handshake hello_request <-- this should be finished
output stream write:

16
fe fd
00 01
00 00 00 00 00 00
00 30
00
0100000000000037c2d8db838d0ec23b49682f7807834921ba1e712740f97a32974c5edf7a275393fe2381f405439c