openssl pkeyutl error Can't set parameter rsa_padding_mode:oaep when decrypting RSA-OAEP ciphertext

[licuser]

Hi,
Using OpenSSL 3.0 and Tpm2 Tools version="5.5"
I am facing a problem when decrypting an RSA-OAEP encrypted data with SHA1. I got error pkeyutl: Can't set parameter "rsa_padding_mode:oaep": as described below:
wrap primary key creation
tpm2_createprimary -C o \ -g sha256 \ -G rsa \ -p $PASS \ -c enroll_rsa.ctx HANDLE=$(tpm2_evictcontrol -c enroll_rsa.ctx | cut -d ' ' -f 2 | head -n 1)
keypair creation
openssl genpkey -provider tpm2 -propquery '?provider=tpm2' \ -algorithm RSA \ -pkeyopt bits:2048 \ -pkeyopt parent:${HANDLE} \ -pkeyopt parent-auth:$PASS \ -pkeyopt user-auth:$USER_PASS \ -out machine.sk.pem
Encrypt data
openssl pkeyutl -encrypt -inkey machinepubkey.pem -pubin -in msg.txt -out msg.enc -pkeyopt rsa_padding_mode:oaep -pkeyopt rsa_oaep_md:sha256
decrypt data
TPM2OPENSSL_PARENT_AUTH=PARENTPASSPHRASE openssl pkeyutl -provider tpm2 -provider base -propquery '?provider=tpm2' -inkey machine.sk.pem -passin pass:keypassword -decrypt -in msg.enc -out msg2.txt -pkeyopt rsa_padding_mode:oaep -pkeyopt rsa_oaep_md:sha1

PROVIDER INIT
DER DECODER DECODE
DER DECODER DECODE
TSS2 DECODER DECODE 0x87
TSS2 DECODER LOAD parent: persistent 0x81000000
TSS2 DECODER DECODE 0x87
TSS2 DECODER LOAD parent: persistent 0x81000000
TSS2 DECODER DECODE found RSA
RSA LOAD
RSA GET_PARAMS [ bits security-bits max-size ]
RSA HAS 1
DECRYPT INIT
DECRYPT SET_CTX_PARAMS [ pad-mode ]
pkeyutl: Can't set parameter "rsa_padding_mode:oaep":
RSA FREE
PROVIDER TEARDOWN

and if I remove padding keyopt from command:
TPM2OPENSSL_PARENT_AUTH=PARENTPASSPHRASE openssl pkeyutl -provider tpm2 -provider base -propquery '?provider=tpm2' -inkey /mnt/licpart/enrollement/machine.sk.pem -passin pass:keypassword -decrypt -in msg.enc -out msg2.txt

PROVIDER INIT
DER DECODER DECODE
DER DECODER DECODE
TSS2 DECODER DECODE 0x87
TSS2 DECODER LOAD parent: persistent 0x81000000
TSS2 DECODER DECODE 0x87
TSS2 DECODER LOAD parent: persistent 0x81000000
TSS2 DECODER DECODE found RSA
RSA LOAD
RSA GET_PARAMS [ bits security-bits max-size ]
RSA HAS 1
DECRYPT INIT
DECRYPT
WARNING:esys:/var/tmp/portage/app-crypt/tpm2-tss-4.0.1/work/tpm2-tss-4.0.1/src/tss2-esys/api/Esys_RSA_Decrypt.c:305:Esys_RSA_Decrypt_Finish() Received TPM Error
ERROR:esys:/var/tmp/portage/app-crypt/tpm2-tss-4.0.1/work/tpm2-tss-4.0.1/src/tss2-esys/api/Esys_RSA_Decrypt.c:102:Esys_RSA_Decrypt() Esys Finish ErrorCode (0x00000084)
Public Key operation error
40F76343C77F0000:error:40000012:tpm2:decrypt_message:cannot decrypt:src/tpm2-provider-asymcipher-rsa.c:81:132 tpm:handle(unk):value is out of range or is not correct for the context
RSA FREE
PROVIDER TEARDOWN

Any help please?

[JuergenReppSIT]

@gotthardp Any idea?

[gotthardp]

Yeah, the OAEP support is not implemented, see tpm2-software/tpm2-openssl#89.

[gotthardp]

I just added support for the OAEP padding, so if you build the latest tpm2-openssl (master branch), your script may work.

[licuser]

Perfect, it works fine now!
Thank you