Loading libpkcs11.so from program with statically compiled OpenSSL?

[barower]

I'm trying to set up secure boot signing for Rock 5B board using PKCS#11 and this program: https://github.com/rockchip-linux/rkbin/blob/master/tools/rk_sign_tool

Unfortunately, this program is closed-source and has little to no documentation. All I know is that it has some version of OpenSSL compiled statically and is configured by this file. It is possible to tell it to use chosen OpenSSL engine by setting those lines as follows:

...
using_hsm=true
hsm_engine_id=pkcs11
hsm_private_key_id='pkcs11:token=SOME_TOKEN;object=SOME_PRIVKEY'
hsm_public_key_id='pkcs11:token=SOME_TOKEN;object=SOME_PUBKEY'

Also, the program expects openssl.cnf to be in the same directory as the binary. This is my configuration:

openssl_conf = openssl_init

[openssl_init]
engines=engine_section

[engine_section]
pkcs11 = pkcs11_section

[pkcs11_section]
engine_id = pkcs11
dynamic_path = /usr/lib/x86_64-linux-gnu/engines-3/libpkcs11.so
MODULE_PATH = /opt/ultimaco/lib/libcs_pkcs11_R3.so
VERBOSE = EMPTY
init = 0

The problem is, that in Ubuntu 22.04 (and most likely in any other Linux distro) libpkcs11.so is dynamically linked to my distro's OpenSSL. I've found that out after compiling libp11 myself with debug symbols and hooking it up to gdb, as it fails exactly here: https://github.com/OpenSC/libp11/blob/master/src/eng_front.c#L92

From my understanding this is a wrong approach, because I'm basically mixing two versions of OpenSSL now, each of different version and with it's own set of static variables etc.

But what could be a good approach here? Is there any way to convince libpkcs11.so to somehow use symbols that are already available, instead of those from my distro's libcrypto.so? I've tried configuring libp11 with LDFLAGS="-Wl,--exclude-libs,all", but the behaviour didn't change. Maybe the problem could be hidden completely elsewhere?

[mtrojnar]

I don't think OpenSSL's engine API allows for loading engines into a statically linked executable. This is not specific to libp11. Please consider opening an issue on https://github.com/openssl/openssl/ instead. Make sure to check whether there is an existing (possibly already closed) issue about it.

[barower]

Thanks! I've looked there and it seems like compiling libp11 without linking it to libcrypto.so in theory could do the trick: openssl/openssl#11294 (comment) openssl/openssl#11294 (comment)

If I'm not mistaken it would not be trivial to do so however, at least without some gymnastics with autotools

[mtrojnar]

This comment describes a very special theoretical scenario where your statically linked application exposes all the OpenSSL symbols that are needed by your engine. I don't think your application does that, but it doesn't hurt to check.

Also, if you read further comments, nobody seems to have succeeded with a practical implementation of this theoretical scenario.

[mtrojnar]

Good news: rk_sign_tool appears to use OpenSSL 3.0.2 and export its symbols. It could work with our pkcs11 engine compiled against OpenSSL 3.0.2.

⚡ strings rk_sign_tool | grep -A 1 openssl-version
openssl-version
3.0.2
⚡ nm rk_sign_tool | grep ENGINE_ | head
000000000047c340 T ENGINE_add
00000000005c1a10 T ENGINE_add_conf_module
000000000047c7a0 T ENGINE_by_id
000000000047c7a0 t ENGINE_by_id.localalias
00000000005c1e90 T ENGINE_cmd_is_executable
00000000005c1a30 T ENGINE_ctrl
00000000005c1f00 T ENGINE_ctrl_cmd
00000000005c2010 T ENGINE_ctrl_cmd_string
000000000047b780 T ENGINE_finish
000000000047bab0 T ENGINE_free

[barower]

This comment describes a very special theoretical scenario where your statically linked application exposes all the OpenSSL symbols that are needed by your engine. I don't think your application does that, but it doesn't hurt to check.

Funnily enough, this could be the case here:

$ nm rk_sign_tool | grep ENGINE_ | wc -l
124
$ nm rk_sign_tool | grep EVP_ | wc -l
852
$ nm rk_sign_tool | grep ERR_ | wc -l
48
$ nm rk_sign_tool | grep UI_ | wc -l
65
$ nm rk_sign_tool | grep BN_ | wc -l
188
$ nm rk_sign_tool | grep RSA_ | wc -l
122
$ nm rk_sign_tool | grep EC_  | wc -l
272
$ nm rk_sign_tool | grep CRYPTO_ | wc -l
89
$ nm rk_sign_tool | grep OBJ_ | wc -l
33
$ nm rk_sign_tool | grep BIO_ | wc -l
155
$ nm rk_sign_tool | grep X509_ | wc -l
644
$ nm rk_sign_tool | grep BUF_ | wc -l
8
$ strings rk_sign_tool | grep "openssl-version" -C1
OSSL_provider_init
openssl-version
3.0.2

There are lots of symbols, however I'm aware that it might not be everything, depending e.g. on compilation flags.

I don't feel comfortable around autotools enough to force libp11 to not link libcrypto, but I'll try looking up some clever method anyways

Edit: I've just noticed your comment above, I see we've come to similar conclusions

[mtrojnar]

I don't feel comfortable around autotools enough to force libp11 to not link libcrypto, but I'll try looking up some clever method anyways

I guess the dynamic loader should not search for symbols that are already available. If I'm right, no special special linker tricks should be needed. Just make sure to use the same version of OpenSSL (3.0.2) for building libp11.

[barower]

After quick looking I've noticed that OpenSSL 3.0.2 is already the default on my Ubuntu 22.04, and this is what I've been compiling libp11 against so far

$ ldd pkcs11.so
        linux-vdso.so.1 (0x00007ffd205b2000)
        libcrypto.so.3 => /lib/x86_64-linux-gnu/libcrypto.so.3 (0x00007b4261600000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007b4261200000)
        /lib64/ld-linux-x86-64.so.2 (0x00007b4261b14000)
$ strings /lib/x86_64-linux-gnu/libcrypto.so | grep "^OpenSSL 3"
OpenSSL 3.0.2 15 Mar 2022

I'll try checking in debugger where do OpenSSL calls go when called from pkcs11.so

[barower]

I've set breakpoint at this line, stepped few instructions and landed at address 0x7ffff7bbcf70, where

(gdb) info files
...
        0x0000000000402000 - 0x00000000007fe034 is .text
...
        0x00007ffff7ab4000 - 0x00007ffff7d10e22 is .text in /lib/x86_64-linux-gnu/libcrypto.so.3

So unfortunately I land on dynamically loaded libcrypto.so. I'm researching this topic further

[barower]

I've tried to recreate roughly what rk_sign_tool would do:

#include <stdio.h>
#include <openssl/engine.h>

int main(int argc, char *argv[]) {
    ENGINE *engine;

    if (argc != 2) {
        printf("Usage: %s <engine_name>\n", argv[0]);
        return 1;
    }
    
    engine = ENGINE_by_id(argv[1]);
    if (!engine) {
        printf("Failed to load engine: %s\n", argv[1]);
        return 1;
    }

    if (!ENGINE_init(engine)) {
        printf("Failed to initialize engine: %s\n", ENGINE_get_id(engine));
        ENGINE_free(engine);
        return 1;
    }

    if (!ENGINE_set_default_RAND(engine)) {
        printf("Failed to set engine as default for random number generation.\n");
        ENGINE_free(engine);
        return 1;
    }

    unsigned char rand_bytes[16];
    if (!RAND_bytes(rand_bytes, sizeof(rand_bytes))) {
        printf("Failed to generate random bytes.\n");
        ENGINE_free(engine);
        return 1;
    }

    printf("Random bytes generated using engine %s:\n", ENGINE_get_id(engine));
    for (int i = 0; i < sizeof(rand_bytes); i++) {
        printf("%02x", rand_bytes[i]);
    }
    printf("\n");

    ENGINE_free(engine);

    return 0;
}

I compile that program with following command:

gcc -g -O0 hsm_emulator.c path/to/libcrypto.a

Where libcrypto.a is a result of compiling OpenSSL from branch openssl-3.0.2 with debugging symbols enabled. This way I get a binary that gives exact same results when i run commands from one of my previous commands on it.

I run it and it works even with my distro's pkcs11.so!

OPENSSL_ENGINES=/usr/lib/x86_64-linux-gnu/engines-3 ./a.out pkcs11
Random bytes generated using engine pkcs11:
67810c56a7f30523d0a38f1630afa482

So what could be the problem with rk_sign_tool? After few debugging sessions and comparing behavior of both programs I've noticed that in my original case the program failed at acquiring following lock:
https://github.com/openssl/openssl/blob/dc9bc6c8e1bd329ead703417a2235ab3e97557ec/crypto/ex_data.c#L41-L47

static EX_CALLBACKS *get_and_lock(OSSL_EX_DATA_GLOBAL *global, int class_index,
                                  int read)
{
    ...
    
    if (global->ex_data_lock == NULL) {
        /*
         * If we get here, someone (who?) cleaned up the lock, so just
         * treat it as an error.
         */
         return NULL;
    }
    
    ...

In rk_sign_tool's case, this lock wasn't actually cleaned up, because it wasn't initialized in the first place! OSSL_EX_DATA_GLOBAL *global is a part of OSSL_LIB_CTX received in both cases exactly here: https://github.com/openssl/openssl/blob/dc9bc6c8e1bd329ead703417a2235ab3e97557ec/crypto/context.c#L408 however after examining memory at default_context_int:

• in rk_sign_tool, OSSL_LIB_CTX default_context_int's values are all zeroes/NULLs, including the lock itself
• in my program, OSSL_LIB_CTX default_context_int is initialized with meaningful values. That initialization happens somewhere during engine acquisition at ENGINE_by_id(argv[1])

My first guess would be that this behavior depends on how symbols are resolved and as a result, where does the default OSS_LIB_CTX gets initialized. There is a potential to make it work without meddling with symbols in libp11, as I managed to do so with my own program. This could be a bug in rk_sign_tool, or maybe something else I didn't take into account. Sadly I've ran out of R&D hours, but thanks for the support so far!

[mtrojnar]

Yes, the application and the engine must use the same library. Compiling them against the same version is the essential first step, but then they need to be linked to the same instance, and not only the same version. With two separate libraries, at least one of them won't get properly initialized. Also, passing pointers to objects initialized with one library and using them in another library will very likely break the dependencies assumed by those libraries.