CMS Decryption fails randomly with BouncyCastle encrypted

[jcarnus]

We are using the Java BouncyCastle lib to create CMS and sign it. We use openssl to decrypt it. This is working pretty well but randomly a CMS cannot be decrypted using OpenSSL but works fine with BouncyCastle.

We tested with various version of BC and OpenSSL and the error is the same .
OpenSSL error is :
Error decrypting CMS structure
22472:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:crypto\evp\evp_enc.c:612:

We have the CMS and private key to replicate the issue but this cannot be communicated privately for security purpose.

[t8m]

What are the algorithms used in the CMS file? This sounds like some encryption key or plaintext padding discrepancy between BC and OpenSSL.

[jcarnus]

We are using the AES256-CBC, with X509 certificates to encrypt for recipientd. One thing I can notice is that Openssl do produces an output file, and then later fails with error mentionned below. If I compare the beginning of the file, the content is really not similar to the expected one. How we can debug this in detail ?

[jcarnus]

I've conducted additional test using .Net class EnvelopedCms (and then Decode(data) and then Decrypt(X509Certificate2Collection) and I am able to sucessfully decrypt the file.

[t8m]

If the content of the output file is completely different i.e. gibberish, it means the symmetric encryption key is broken.

By any chance are there multiple recipient certificates in the encrypted CMS file?

[t8m]

Also what openssl version is being used?

[jcarnus]

Test are conducted with OpenSSL 1.1.1q 5 Jul 2022

First few char of the first line of the expected file, also obtained with BC or .Net

Invalid one with Openssl:

Also, the invalid file has almost the same size than the good one.

and yes, they are multiple recipient certificates used.

[jcarnus]

I've just found that setting the -recip in the command line solved the issue. What could cause this issue when not giving the -recip

[t8m]

Yes, you must add -recip if there are multiple recipients to be able to decrypt messages without such random errors. Due to API hardening the RSA decryption does not return error but actually decrypts to a random key, otherwise Bleichenbacher attacks are possible against the private key due to timing or other side channel leaks.

[jcarnus]

Thanks, do you have an idea if such behavior was in past release of openssl ?

[t8m]

No, and this looks like you are using some vendor-patched 1.1.1q version that has this security hardening patch backported into.