Signing scheme RSAPSS is not detected consistently

[akorb]

Tested with 5.6-rc0

tpm2 createprimary -Q -C e -G rsa2048:rsapss-sha256:null -c ek.ctx -o ek.pub -a 'fixedtpm|fixedparent|sensitivedataorigin|userwithauth|sign|restricted'

# This fails with "unsupported or incompatible scheme", since it can't derive RSAPSS from ek.ctx
tpm2 quote -Q -c ek.ctx -g sha256 -l sha1:0,1,2,3 -o quote.pcrs -m quote.msg -s quote.sig

# Provide scheme explicitly
tpm2 quote -Q -c ek.ctx -g sha256 -l sha1:0,1,2,3 -o quote.pcrs -m quote.msg -s quote.sig --scheme=rsapss 

# Signature verification fails
tpm2 checkquote -u ek.pub -m quote.msg -s quote.sig -f quote.pcrs

# Manual signature verification succeeds, however
tpm2 verifysignature -c ek.ctx -s quote.sig -m quote.msg

With tpm2_print -t TPM2B_PUBLIC ek.pub, the RSAPSS-sha256 scheme indicator is printed just fine

name-alg:
  value: sha256
  raw: 0xb
attributes:
  value: fixedtpm|fixedparent|sensitivedataorigin|userwithauth|restricted|sign
  raw: 0x50072
type:
  value: rsa
  raw: 0x1
exponent: 65537
bits: 2048
scheme:
  value: rsapss
  raw: 0x16
scheme-halg:
  value: sha256
  raw: 0xb
sym-alg:
  value: null
  raw: 0x10
sym-mode:
  value: (null)
  raw: 0x0
sym-keybits: 0
rsa: ac11b3697a7d0a3de9811e41659f3fe1a989cbf83665341940238f4f13522abc000682b5f96cc4b96853a9ce930c513d8420abab558025eaae281090a1309f97675c22c7fc0560585b16fc40bd8c99d9008b3eaa9cb0c9ccc1eba4ae6cba7947e0bf6e505929c38b1e236328313346e3816a95a24da3e3fa475f428f5f6d9b350ebd1ecf94c59776237864ac0d0a877d3d68a91ff1390afd95cb281398b0ec849db8c788802e457325259dfda5c8f0450d19b712f355c7400a00de3c1ddc5cc1113e00222c024c6233e348abb227525ef3fd39af458a439bc2236436cc9e5140a7f1f1011ca81e1d0f1b60ba6fc842cdcdba63edf76e5fdba549b66c0d031465

tpm2 checkquote doesn't let me set the signing scheme explicitly unfortunately (#2053 (comment)). Which would be good for a workaround.

I'm not sure whether I'm overriding the signing scheme the checkquote and quote tool use somehow, or if there's a problem in the signing scheme detection code.

[JuergenReppSIT]

@akorb The command seems to be work in progress. As a first step I have added padding guessing.