Kernel >= 5.11 (5.12.9) how to load driver?

[vans163]

Looking at is_sgx_available

bool sgx_driver_loaded() {
    // /dev/isgx is for LKM version, /dev/sgx is for in-kernel support.
    return file_exists("/dev/isgx") || file_exists("/dev/sgx");
}

./is_sgx_available 
SGX supported by CPU: true
SGX1 (ECREATE, EENTER, ...): true
SGX2 (EAUG, EACCEPT, EMODPR, ...): true
Flexible Launch Control (IA32_SGXPUBKEYHASH{0..3} MSRs): true
SGX extensions for virtualizers (EINCVIRTCHILD, EDECVIRTCHILD, ESETCONTEXT): true
Extensions for concurrent memory management (ETRACKC, ELDBC, ELDUC, ERDINFO): true
CET enclave attributes support (See Table 37-5 in the SDM): false
Key separation and sharing (KSS) support (CONFIGID, CONFIGSVN, ISVEXTPRODID, ISVFAMILYID report fields): true
Max enclave size (32-bit): 0x80000000
Max enclave size (64-bit): 0x800000000000
EPC size: 0x5cc0000
SGX driver loaded: false
SGX PSW/libsgx installed: false
AESMD running: false

ls /dev | grep sgx

sgx_enclave
sgx_provision

uname -a

5.12.9-051209-generic

My understanding is that kernel 5.11 carries all the needed kmods to run graphene-sgx, but it seems like it wants the driver, when I try to run the example in different ways:

graphene-sgx helloworld
error: Cannot open device /dev/sgx_enclave. Please make sure the Intel SGX kernel module is loaded.
error: load_enclave() failed with error -13

sudo graphene-sgx helloworld
error: Cannot map enclave pages 1
error: Loading enclave binary failed: -13
error: load_enclave() failed with error -13

graphene-direct helloworld
error: Using insecure argv source. Graphene will continue application execution, but this configuration must not be used in production!
Hello world!

I dont see SGX in lsmod. Guessing I need to install https://github.com/intel/SGXDataCenterAttestationPrimitives/tree/master/driver/linux but its in a transitional stage so soon this will be upstreamed aswell?

But the changelog here says

V1.41
Sync with upstream patch v41, the last one before merged to mainline 5.11 release.

When I try to make it

cd SGXDataCenterAttestationPrimitives/driver/linux
make

Makefile:24: *** Can't install DCAP SGX driver with inkernel SGX support.  Stop.

[mkow]

Oops, seems we forgot to update this tool when SGX drivers got upstreamed. Please check if #2438 fixes the problem for you.

From the description you provided it seems that your system is fine and should be able to run SGX, it's just our script which is outdated :)

[vans163]

Well this makes it just return true for sgx_driver_loaded but the helloworld example app is having the same errors. I am guessing based on what I posted it might be a problem with the helloworld example not updated for kernel >= 5.11?

[mkow]

Ah, I missed that there is one more issue in your description.

error: Cannot open device /dev/sgx_enclave. Please make sure the Intel SGX kernel module is loaded.

This is most likely caused by quite bad support for SGX in current Linux distros. AFAIR by default this device is accessible only by root, so you just need to chmod it properly. To my best knowledge doing this shouldn't be a security issue, as enclaves doesn't give additional permissions to their users. The only problem may be increasing the attack surface by exposing more kernel interfaces to the users.

[vans163]

So even as root (or sudo) / giving perms to this dev to another user.

sudo graphene-sgx helloworld

debug: Token file: helloworld.token
debug: Read dummy DCAP token
debug: enclave created:
debug:     base:           0x0000000000000000
debug:     size:           0x0000000008000000
debug:     misc_select:    0x00000000
debug:     attr.flags:     0x0000000000000007
debug:     attr.xfrm:      0x00000000000002e7
debug:     ssa_frame_size: 4
debug:     isv_prod_id:    0x00000000
debug:     isv_svn:        0x00000000
debug: adding pages to enclave: 0x7ffe000-0x8000000 [REG:R--] (manifest) measured
debug: adding pages to enclave: 0x7f7e000-0x7ffe000 [REG:RW-] (ssa) measured
debug: adding pages to enclave: 0x7f6e000-0x7f7e000 [TCS:---] (tcs) measured
debug: adding pages to enclave: 0x7f5e000-0x7f6e000 [REG:RW-] (tls) measured
debug: adding pages to enclave: 0x7f1e000-0x7f5e000 [REG:RW-] (stack) measured
debug: adding pages to enclave: 0x7ede000-0x7f1e000 [REG:RW-] (stack) measured
debug: adding pages to enclave: 0x7e9e000-0x7ede000 [REG:RW-] (stack) measured
debug: adding pages to enclave: 0x7e5e000-0x7e9e000 [REG:RW-] (stack) measured
debug: adding pages to enclave: 0x7e1e000-0x7e5e000 [REG:RW-] (stack) measured
debug: adding pages to enclave: 0x7dde000-0x7e1e000 [REG:RW-] (stack) measured
debug: adding pages to enclave: 0x7d9e000-0x7dde000 [REG:RW-] (stack) measured
debug: adding pages to enclave: 0x7d5e000-0x7d9e000 [REG:RW-] (stack) measured
debug: adding pages to enclave: 0x7d1e000-0x7d5e000 [REG:RW-] (stack) measured
debug: adding pages to enclave: 0x7cde000-0x7d1e000 [REG:RW-] (stack) measured
debug: adding pages to enclave: 0x7c9e000-0x7cde000 [REG:RW-] (stack) measured
debug: adding pages to enclave: 0x7c5e000-0x7c9e000 [REG:RW-] (stack) measured
debug: adding pages to enclave: 0x7c1e000-0x7c5e000 [REG:RW-] (stack) measured
debug: adding pages to enclave: 0x7bde000-0x7c1e000 [REG:RW-] (stack) measured
debug: adding pages to enclave: 0x7b9e000-0x7bde000 [REG:RW-] (stack) measured
debug: adding pages to enclave: 0x7b5e000-0x7b9e000 [REG:RW-] (stack) measured
debug: adding pages to enclave: 0x7b4e000-0x7b5e000 [REG:RW-] (sig_stack) measured
debug: adding pages to enclave: 0x7b3e000-0x7b4e000 [REG:RW-] (sig_stack) measured
debug: adding pages to enclave: 0x7b2e000-0x7b3e000 [REG:RW-] (sig_stack) measured
debug: adding pages to enclave: 0x7b1e000-0x7b2e000 [REG:RW-] (sig_stack) measured
debug: adding pages to enclave: 0x7b0e000-0x7b1e000 [REG:RW-] (sig_stack) measured
debug: adding pages to enclave: 0x7afe000-0x7b0e000 [REG:RW-] (sig_stack) measured
debug: adding pages to enclave: 0x7aee000-0x7afe000 [REG:RW-] (sig_stack) measured
debug: adding pages to enclave: 0x7ade000-0x7aee000 [REG:RW-] (sig_stack) measured
debug: adding pages to enclave: 0x7ace000-0x7ade000 [REG:RW-] (sig_stack) measured
debug: adding pages to enclave: 0x7abe000-0x7ace000 [REG:RW-] (sig_stack) measured
debug: adding pages to enclave: 0x7aae000-0x7abe000 [REG:RW-] (sig_stack) measured
debug: adding pages to enclave: 0x7a9e000-0x7aae000 [REG:RW-] (sig_stack) measured
debug: adding pages to enclave: 0x7a8e000-0x7a9e000 [REG:RW-] (sig_stack) measured
debug: adding pages to enclave: 0x7a7e000-0x7a8e000 [REG:RW-] (sig_stack) measured
debug: adding pages to enclave: 0x7a6e000-0x7a7e000 [REG:RW-] (sig_stack) measured
debug: adding pages to enclave: 0x7a5e000-0x7a6e000 [REG:RW-] (sig_stack) measured
debug: adding pages to enclave: 0x3642000-0x3684000 [REG:R-X] (code) measured
error: Cannot map enclave pages 1
error: Loading enclave binary failed: -13
error: load_enclave() failed with error -13

It seems

#ifdef SGX_DCAP
    /* ask Intel SGX driver to actually mmap the added enclave pages */
    uint64_t mapped = INLINE_SYSCALL(mmap, 6, addr, size, prot, MAP_FIXED | MAP_SHARED,
                                     g_isgx_device, 0);
    if (IS_ERR_P(mapped)) {
        log_error("Cannot map enclave pages %ld\n", ERRNO_P(mapped));
        return -EACCES;
    }

That I compiled it with SGX_DCAP defined, but I am using inkernel driver, maybe some mixup here?

Guessing the 1 here is EPERM 1 Operation not permitted ?

Some kind of permission error (maybe apparmor or some security profile, tho I turned apparmor off and samething), I am testing on Ubuntu 21.04, with a i7-1065G7

Thinking its this, https://askubuntu.com/questions/1233973/what-happened-to-mmap-in-20-04

[vans163]

K solved, so on ubuntu 20.04 and up need to remount /dev and remove noexec. I wonder if there is a better fix for this?

mount -o remount,exec /dev

EDIT:
Probably this https://graphene.readthedocs.io/en/latest/sgx-intro.html is best to reference too to change udev rules.

[dimakuv]

@vans163 I didn't encounter this "remount /dev". Where did you find this suggestion? Why does it help?

After quick googling, I found this: https://unix.stackexchange.com/questions/581928/is-it-safe-to-add-noexec-mount-option-in-dev. And it looks like Debian/Ubuntu indeed now mounts with noexec by default. But I don't see how this could affect the Intel SGX driver and Graphene... Strange.

[mkow]

This is interesting, but I think it's only on 21.04 and up. We do testing on 20.04 and it works there, I even checked on my instance:

udev on /dev type devtmpfs (rw,nosuid,relatime,size=<snip>k,nr_inodes=<snip>,mode=755)

CC: @woju, seems 21.04 broke SGX support altogether?

[jinengandhi-intel]

I was trying to setup a new machine with 5.11 upstream kernel and Ubuntu 20.04 and I faced this same issue even there, was able to resolve with mount -o remount,exec /dev
Same issue is also seen on 5.12 upstream kernel.

[dimakuv]

Just a quick note on this, since we hit this today. Actually there are two separate issues:

• The SGX driver /dev/sgx_enclave has wrong permissions. Typically these are 700 ("only root can read/write/execute"). You can check your permissions on this file with ls -la /dev/sgx_enclave. In reality, the permissions should be set to something like 660 and /dev/sgx_enclave must be rewired to group sgx, and your user must be added to this group sgx (so in the end it reads as "root can read and write, other users in group sgx can also read and write"). This is explained in our docs: https://graphene.readthedocs.io/en/latest/sgx-intro.html#linux-kernel-drivers

• The whole /dev directory is mounted with noexec attribute. A workaround is to remount it via mount -o remount,exec /dev (removing the noexec attribute). It's unclear why this is needed because Graphene doesn't request the EXEC permission when opening the driver, see: https://github.com/oscarlab/graphene/blob/4cb98219d8e302055587a8952c1102415a72ba42/Pal/src/host/Linux-SGX/sgx_framework.c#L34

So one should check for both these issues. These issues are there until modern Linux OS distributions are shipped with correct installation scripts for the Linux SGX driver.

[mkow]

Yup, this is basically distros breaking SGX driver in their default setups. And they don't ship any usermode SGX daemons yet, so they probably don't even know that they've broken something.

[dimakuv]

The issue itself was resolved, closing.

[mkow]

Was it really resolved? This seems to be quite a big problem for SGX ecosystem, I'd say we need to either document this or ask Ubuntu devs to fix this.

[Nada-Bu]

I face the same problem. Using this command: mount -o remount,exec /dev, solved it.
Thanks