Can't mount encrypted partition when LTO is enabled

[hadess]

Originally reported as https://bugzilla.redhat.com/show_bug.cgi?id=1950208

Using the current git HEAD version (ee71aa5)

$ sudo apfs-fuse /dev/sdd2 /mnt -d 31
Device /dev/sdd2 opened. Size is 16000000000
Mounting xid different from NXSB at 0 (xid = 36). xid = 36
Mounting xid 36
omap: oid=1027 xid=36 flags=0 size=0 paddr=1027
omap: oid=1028 xid=36 flags=0 size=0 paddr=1028
starting LoadKeybag @ 20003
Initialization of KeyManager failed.
Unable to load container.

This seems to be caused by Fedora enabling LTO by default in its builds:
https://fedoraproject.org/wiki/LTOByDefault

Building with LTO generate quite a few new warnings compared to without LTO, including a fair bunch of signedness issues, which are probably symptoms of places that LTO will eventually modify and thus break.

• Build without LTO
• Build with LTO

[sgan81]

Stupid aliasing rules ... anyway, it should be fixed now, at the expense of encryption speed ...

[sgan81]

So, after a few more tests, I found the following:

• The original code works if -flto is specified alongside with -fno-strict-alias
• LTO with clang works fine

Apparently, the reason was that I had done some pointer casting in aes.Encrypt and aes.Decrypt (void * to uint32_t *) in order to speed up things. GCC was of the opinion though that aes.Encrypt/Decrypt did not write anything, and that the uint32_t * did not alias the void *, and therefore the whole call to aes.Decrypt could be optimized out ...

So I don't really know what's better ... having somewhat slower code, or using -fno-strict-alias when compiling with GCC and LTO ...

[RJVB]

On Wednesday January 04 2023 18:42:33 Simon Gander wrote: So I don't really know what's better ... having somewhat slower code, or using -fno-strict-alias when compiling with GCC and LTO ...

I'd say the latter because that's a condition you can check for and handle at configure time. And the flag need only be passed when compiling that single file. FWIW, doesn't `-fno-strict-alias` reenable us to do the things we could do in C, back when we were still allowed to know better than the compiler and *write* optimised code?