Verify misbehaves when multiple signatures exists

[pontus]

apptainer supports multiple signatures, but verification does not seem to work as expected for these, instead adding a new signature will break verification for the previously existing signature.

Version of Apptainer

from git commit 8028a38 (17 Apr)
1.3.0+170-g8028a3889-dirty

Steps to reproduce this behavior

pontus@lalage:~/Utveckling/apptainer/builddir$ ./apptainer build /tmp/test.sif docker://alpine  
INFO:    Starting build...                                                                     
Copying blob 4abcf2066143 skipped: already exists                                              
Copying config bc4e4f7999 done   |                                                                                                                                                            
Writing manifest to image destination                                                                                                                                                         
2024/04/18 09:29:20  info unpack layer: sha256:4abcf20661432fb2d719aaf90656f55c287f8ca915dc1c92ec14ff61e67fbaf8
INFO:    Creating SIF file...                                                                  
INFO:    Build complete: /tmp/test.sif             
pontus@lalage:~/Utveckling/apptainer/builddir$ openssl genrsa  -out key1 1024                   
pontus@lalage:~/Utveckling/apptainer/builddir$ openssl genrsa  -out key2 1024        
pontus@lalage:~/Utveckling/apptainer/builddir$ openssl rsa -in key1 -pubout > key1.pub
writing RSA key
pontus@lalage:~/Utveckling/apptainer/builddir$ openssl rsa -in key2 -pubout > key2.pub          
writing RSA key
pontus@lalage:~/Utveckling/apptainer/builddir$ ./apptainer sign  --key key1 /tmp/test.sif 
INFO:    Signing image with key material from 'key1'
INFO:    Signature created and applied to image '/tmp/test.sif'
pontus@lalage:~/Utveckling/apptainer/builddir$ ./apptainer sign  --key key2 /tmp/test.sif       
INFO:    Signing image with key material from 'key2'
INFO:    Signature created and applied to image '/tmp/test.sif'
pontus@lalage:~/Utveckling/apptainer/builddir$ ./apptainer verify --key key1.pub /tmp/test.sif 

Expected behavior

pontus@lalage:~/Utveckling/apptainer/builddir$ ./apptainer verify --key key1.pub /tmp/test.sif 
INFO:    Verifying image with key material from 'key1.pub'
Objects verified:
ID  |GROUP   |LINK    |TYPE
------------------------------------------------
1   |1       |NONE    |Def.FILE
2   |1       |NONE    |JSON.Generic
3   |1       |NONE    |JSON.Generic
4   |1       |NONE    |FS
INFO:    Verified signature(s) from image '/tmp/test.sif'
pontus@lalage:~/Utveckling/apptainer/builddir$ echo $?
0
pontus@lalage:~/Utveckling/apptainer/builddir$ 

Actual behavior

pontus@lalage:~/Utveckling/apptainer/builddir$ ./apptainer verify --key key1.pub /tmp/test.sif 
INFO:    Verifying image with key material from 'key1.pub'
Objects verified:
ID  |GROUP   |LINK    |TYPE
------------------------------------------------
1   |1       |NONE    |Def.FILE
2   |1       |NONE    |JSON.Generic
3   |1       |NONE    |JSON.Generic
4   |1       |NONE    |FS

Error encountered during signature verification: signature object 6 not valid: dsse: verify envelope failed: accepted signatures do not match threshold, Found: 0, Expected 1
FATAL:   Failed to verify container: integrity: signature object 6 not valid: dsse: verify envelope failed: accepted signatures do not match threshold, Found: 0, Expected 1
pontus@lalage:~/Utveckling/apptainer/builddir$ echo $?
255
pontus@lalage:~/Utveckling/apptainer/builddir$ ./apptainer verify --key key2.pub /tmp/test.sif 
INFO:    Verifying image with key material from 'key2.pub'

Error encountered during signature verification: signature object 5 not valid: dsse: verify envelope failed: accepted signatures do not match threshold, Found: 0, Expected 1
FATAL:   Failed to verify container: integrity: signature object 5 not valid: dsse: verify envelope failed: accepted signatures do not match threshold, Found: 0, Expected 1
pontus@lalage:~/Utveckling/apptainer/builddir$ echo $?
255
pontus@lalage:~/Utveckling/apptainer/builddir$ 

If using the public key corresponding to the first signature, the user gets a FATAL error about the signature object for the second object not being possible to verify, but I'd argue we shouldn't care about that (adding a signature shouldn't break verification of previous).

It is somewhat communicated to the users that part of the file do verify, but that's followed by a scare error and the exit code signals an error.

For the case of the key for the second signature, one gets the same error for a different signature object, but it doesn't list any objects being verified. I don't know but my guess is that it doesn't try to look at the correct signature object (6 in the example above) but rather sees that the first it encounters doesn't match the provided key and bails out.

(Exit code matters since this might be used automated and bail out if it fails, e.g. a workflow where "blessed" images are signed and some script tries to verify that by checking that a given image is signed with the relevant key - that will currently break at the stage of verification.

What OS/distro are you running

pontus@lalage:~/Utveckling/apptainer/builddir$ cat /etc/os-release 
PRETTY_NAME="Ubuntu Noble Numbat (development branch)"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo
pontus@lalage:~/Utveckling/apptainer/builddir$ 

How did you install Apptainer

Built from git main.

[dtrudg]

FYI - there was previous discussion of this / related multiple signature issues in SingularityCE at sylabs/singularity#1150

[DrDaveD]

Thanks for the link, DT. That refers to #462 which is also related, but already implemented.

[jrwds]

I originally brought this up. The errors make sense in hindsight. It was not obvious to me that by default Apptainer attempts to verify all signatures at once. I may submit a PR for userdocs again when I discover the final right way to verify all sigs at once (might be a while- I am slow and have other priorities and still have a lot to learn about X509) pointing this out. This probably has little to do with Apptainer, and more to do with me being completely baffled by X509 :^)

I don't think any code changes need to be made. However for a given sif if object 5, 6, 7 have signatures, and only the cert for 7 is provided, it will complain about 5 missing, although 6 is missing too. It might take a lot of work to make checks for all the signatures and report which ones are missing, and I am skeptical it's necessary.

[pontus]

I disagree, I think the the current behaviour doesn't make sense. If there is a use for requiring all signatures to be verified, it should be chosen by a flag.

Even worse, specifically for the directly supplied public key, it didn't (somewhat surprisingily) even work to provide keys for both signatures either.