gramine-sgx-get-token fails to connect to aesmd from within a container #42

jpelkonen · 2022-01-06T22:36:22Z

Description of the problem

This is an existing on-prem (Intel Nuc) system that has been used for graphene-sgx in the past. Our team is now looking to upgrade to gramine, and for the most part everything appears to be working correcly, however, when we attempt build and test container images, the test fails at runtime.

Any help or pointers are appreciated.

Steps to reproduce

gsc commit id: 21b30f2
Run

 ./gsc build -d  --insecure-args ubuntu18.04-bash test/ubuntu18.04-bash.manifest
./gsc sign-image ubuntu18.04-bash  ~/.config/gramine/enclave-key.pem
docker run --privileged --device=/dev/isgx  --device=/dev/gsgx      -v /var/run/aesmd/aesm.socket:/var/run/aesmd/aesm.socket       gsc-ubuntu18.04-bash -c ls

Expected results

Files are listed

Actual results

The following error is printed:

        + find /gramine/meson_build_output/lib -type d -path '*/site-packages'
        + export PYTHONPATH=:/gramine/meson_build_output/lib/python3.6/site-packages
        + PYTHONPATH=:/gramine/meson_build_output/lib/python3.6/site-packages
        ++ find /gramine/meson_build_output/lib -type d -path '*/pkgconfig'
        + export PKG_CONFIG_PATH=:/gramine/meson_build_output/lib/x86_64-linux-gnu/pkgconfig
        + PKG_CONFIG_PATH=:/gramine/meson_build_output/lib/x86_64-linux-gnu/pkgconfig
        + '[' -z '' ']'
        + gramine-sgx-get-token --sig /entrypoint.sig --output /entrypoint.token
        Attributes:
            mr_enclave:  208579237095fcc1358557f23e5903c4bca6d9d36b495a319ba787b497b1e0bc
            mr_signer:   89139782896bcfc6cdc46924b53fdc83b8886ea681230bc5bb77f584bacaf255
            isv_prod_id: 0
            isv_svn:     0
            attr.flags:  0000000000000004
            attr.xfrm:   0000000000000007
            mask.flags:  ffffffffffffffff
            mask.xfrm:   fffffffffff9ff1b
            misc_select: 00000000
            misc_mask:   ffffffff
            modulus:     dd47372296ab46c9351e57564457be48...
            exponent:    3
            signature:   e7327dff8d92bebf22f34a78c864ff4c...
            date:        2022-01-06
        Traceback (most recent call last):
          File "/gramine/meson_build_output/bin/gramine-sgx-get-token", line 20, in <module>
            main() # pylint: disable=no-value-for-parameter
          File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1128, in __call__
            return self.main(*args, **kwargs)
          File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1053, in main
            rv = self.invoke(ctx)
          File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1395, in invoke
            return ctx.invoke(self.callback, **ctx.params)
          File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 754, in invoke
            return __callback(*args, **kwargs)
          File "/gramine/meson_build_output/bin/gramine-sgx-get-token", line 16, in main
            token = get_token(sig, verbose=verbose)
          File "/gramine/meson_build_output/lib/python3.6/site-packages/graminelibos/sgx_get_token.py", line 152, in get_token
            token = connect_aesmd(sig['enclave_hash'], sig['modulus'], sig['attribute_flags'], xfrms)
          File "/gramine/meson_build_output/lib/python3.6/site-packages/graminelibos/sgx_get_token.py", line 89, in connect_aesmd
            raise Exception(f'Failed. (Error Code = {ret_msg.ret.error})')
        Exception: Failed. (Error Code = 6)

Additional information

SGX is installed, aesmd is running (see output from is-sgx-below) and our old graphene-sgx containers function correctly

is-sgx-available
SGX supported by CPU: true
SGX1 (ECREATE, EENTER, ...): true
SGX2 (EAUG, EACCEPT, EMODPR, ...): false
Flexible Launch Control (IA32_SGXPUBKEYHASH{0..3} MSRs): false
SGX extensions for virtualizers (EINCVIRTCHILD, EDECVIRTCHILD, ESETCONTEXT): false
Extensions for concurrent memory management (ETRACKC, ELDBC, ELDUC, ERDINFO): false
CET enclave attributes support (See Table 37-5 in the SDM): false
Key separation and sharing (KSS) support (CONFIGID, CONFIGSVN, ISVEXTPRODID, ISVFAMILYID report fields): false
Max enclave size (32-bit): 0x80000000
Max enclave size (64-bit): 0x1000000000
EPC size: 0x5d80000
SGX driver loaded: true
AESMD installed: true
SGX PSW/libsgx installed: true

The text was updated successfully, but these errors were encountered:

mkow · 2022-01-06T23:42:28Z

Could you check (and paste here) the output of sudo service aesmd status on the host?

dimakuv · 2022-01-07T08:32:21Z

Also could you check the output of dmesg? It may also contain some interesting AESMD messages.

Also, can you show your config.yaml file?

jpelkonen · 2022-01-07T15:17:17Z

Could you check (and paste here) the output of sudo service aesmd status on the host?

Thanks for the quick response. Here's the status:

sudo service aesmd status
● aesmd.service - Intel(R) Architectural Enclave Service Manager
   Loaded: loaded (/lib/systemd/system/aesmd.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2022-01-06 14:37:48 CST; 18h ago
  Process: 4270 ExecStart=/opt/intel/libsgx-enclave-common/aesm/aesm_service (code=exited, status=0/SUCCESS)
  Process: 4269 ExecStartPre=/bin/chmod 0755 /var/run/aesmd/ (code=exited, status=0/SUCCESS)
  Process: 4268 ExecStartPre=/bin/chown -R aesmd:aesmd /var/run/aesmd/ (code=exited, status=0/SUCCESS)
  Process: 4255 ExecStartPre=/bin/mkdir -p /var/run/aesmd/ (code=exited, status=0/SUCCESS)
  Process: 4243 ExecStartPre=/opt/intel/libsgx-enclave-common/aesm/linksgx.sh (code=exited, status=0/SUCCESS)
 Main PID: 4271 (aesm_service)
    Tasks: 4 (limit: 4915)
   CGroup: /system.slice/aesmd.service
           └─4271 /opt/intel/libsgx-enclave-common/aesm/aesm_service

Jan 06 14:37:48 ideanovalicensing systemd[1]: Starting Intel(R) Architectural Enclave Service Manager...
Jan 06 14:37:48 ideanovalicensing systemd[1]: Started Intel(R) Architectural Enclave Service Manager.
Jan 06 14:37:48 ideanovalicensing aesm_service[4271]: [ADMIN]White List update requested
Jan 06 14:37:48 ideanovalicensing aesm_service[4271]: [ADMIN]Platform Services initializing
Jan 06 14:37:48 ideanovalicensing aesm_service[4271]: [ADMIN]Platform Services initialization failed due to DAL error
Jan 06 14:37:48 ideanovalicensing aesm_service[4271]: The server sock is 0x562731c2ee10
Jan 06 14:37:48 ideanovalicensing aesm_service[4271]: [ADMIN]White list update request successful for Version: 108

jpelkonen · 2022-01-07T15:35:08Z

Also could you check the output of dmesg? It may also contain some interesting AESMD messages.

Also, can you show your config.yaml file?

Thanks for the suggestion. Here are my updates:

dmesg  | grep -i aesmd

No output.

cat config.yaml
# Specify the OS distro. Currently tested distros are Ubuntu 18.04 and Ubuntu 20.04.
Distro: "ubuntu:18.04"

# If you're using your own fork and branch of Gramine, specify the GitHub link and the branch name
# below; typically, you want to keep the default values though
Gramine:
    Repository: "https://github.com/gramineproject/gramine.git"
    Branch:     "master"

# Specify the Intel SGX driver installed on your machine (more specifically, on the machine where
# the graminized Docker container will run); there are several variants of the SGX driver:
#
#   - legacy out-of-tree driver: use something like the below values, but adjust the branch name
#         Repository: "https://github.com/01org/linux-sgx-driver.git"
#         Branch:     "sgx_driver_1.9"
#
#   - DCAP out-of-tree driver: use something like the below values
#         Repository: "https://github.com/intel/SGXDataCenterAttestationPrimitives.git"
#         Branch:     "DCAP_1.11 && cp -r driver/linux/* ."
#
#   - DCAP in-kernel driver: use empty values like below
#         Repository: ""
#         Branch:     ""
#
SGXDriver:
    Repository: "https://github.com/01org/linux-sgx-driver.git"
    Branch:     "sgx_driver_2.5"

I just want to reiterate that our old graphene-sgx based containers are working as expected, so it seems aesmd is ok.

dimakuv · 2022-01-07T16:03:37Z

Have you tried normal Gramine? Like described in https://gramine.readthedocs.io/en/latest/quickstart.html (installing from an Ubuntu package) and in https://gramine.readthedocs.io/en/latest/devel/building.html (git-cloning the latest version).

jpelkonen · 2022-01-07T17:11:39Z

Have you tried normal Gramine? Like described in https://gramine.readthedocs.io/en/latest/quickstart.html (installing from an Ubuntu package) and in https://gramine.readthedocs.io/en/latest/devel/building.html (git-cloning the latest version).

We have installed the package gramine-oot
We have also cloned the repo and builth the hello-world:


make SGX=1 SGX_SIGNER_KEY="$HOME"/.config/gramine/enclave-key.pem
cc -Wall -Wextra -O3   -c -o helloworld.o helloworld.c
cc   helloworld.o   -o helloworld
gramine-manifest \
	-Dlog_level=error \
	helloworld.manifest.template helloworld.manifest
gramine-sgx-sign \
	--key /home/int_license/.config/gramine/enclave-key.pem \
	--manifest helloworld.manifest \
	--output helloworld.manifest.sgx
Attributes:
    size:        0x10000000
    thread_num:  4
    isv_prod_id: 0
    isv_svn:     0
    attr.flags:  0x6
    attr.xfrm:   0x3
    misc_select: 0x0
Memory:
    000000000fffd000-0000000010000000 [REG:R--] (manifest) measured
    000000000ffdd000-000000000fffd000 [REG:RW-] (ssa) measured
    000000000ffd9000-000000000ffdd000 [TCS:---] (tcs) measured
    000000000ffd5000-000000000ffd9000 [REG:RW-] (tls) measured
    000000000ff95000-000000000ffd5000 [REG:RW-] (stack) measured
    000000000ff55000-000000000ff95000 [REG:RW-] (stack) measured
    000000000ff15000-000000000ff55000 [REG:RW-] (stack) measured
    000000000fed5000-000000000ff15000 [REG:RW-] (stack) measured
    000000000fec5000-000000000fed5000 [REG:RW-] (sig_stack) measured
    000000000feb5000-000000000fec5000 [REG:RW-] (sig_stack) measured
    000000000fea5000-000000000feb5000 [REG:RW-] (sig_stack) measured
    000000000fe95000-000000000fea5000 [REG:RW-] (sig_stack) measured
    000000000ba71000-000000000babc000 [REG:R-X] (code) measured
    000000000babc000-000000000fe95000 [REG:RW-] (data) measured
    0000000000010000-000000000ba71000 [REG:RWX] (free)
Measurement:
    a46f072128759ed4f0e935f82ed05dc4fd1be9f55e3c382b34a080cb5836ea9c
gramine-sgx-get-token \
	--output helloworld.token --sig helloworld.sig
Attributes:
    mr_enclave:  a46f072128759ed4f0e935f82ed05dc4fd1be9f55e3c382b34a080cb5836ea9c
    mr_signer:   89139782896bcfc6cdc46924b53fdc83b8886ea681230bc5bb77f584bacaf255
    isv_prod_id: 0
    isv_svn:     0
    attr.flags:  0000000000000006
    attr.xfrm:   0000000000000007
    mask.flags:  ffffffffffffffff
    mask.xfrm:   ffffffffffffff1b
    misc_select: 00000000
    misc_mask:   ffffffff
    modulus:     dd47372296ab46c9351e57564457be48...
    exponent:    3
    signature:   b51cbf1e4b95e50f92680c354d84276f...
    date:        2022-01-07

Running as a regular user fails:

gramine-sgx helloworld
error: Permission denied on mapping enclave. You may need to set sysctl vm.mmap_min_addr to zero
error: ECREATE failed in allocating EPC memory: -1
error: Creating enclave failed: -1
error: load_enclave() failed with error -1

With sudo

 sudo gramine-sgx helloworld
-----------------------------------------------------------------------------------------------------------------------
Gramine detected the following insecure configurations:

  - sgx.debug = true                           (this is a debug enclave)

Gramine will continue application execution, but this configuration must not be used in production!
-----------------------------------------------------------------------------------------------------------------------

Hello, world

Thanks again for the help! Truly appreciated

dimakuv · 2022-01-10T08:05:17Z

error: Permission denied on mapping enclave. You may need to set sysctl vm.mmap_min_addr to zero

Gramine gives you this error message. It means that you should do something like this: sudo sysctl vm.mmap_min_addr=0. After this, you won't need to run Gramine as sudo.

jpelkonen · 2022-01-12T15:58:23Z

Gramine gives you this error message. It means that you should do something like this: sudo sysctl vm.mmap_min_addr=0. After this, you won't need to run Gramine as sudo.

Thanks @dimakuv. I can run gramine without sudo, however, the original gsc issue still persist:

gramine-sgx helloworld
-----------------------------------------------------------------------------------------------------------------------
Gramine detected the following insecure configurations:

  - sgx.debug = true                           (this is a debug enclave)

Gramine will continue application execution, but this configuration must not be used in production!
-----------------------------------------------------------------------------------------------------------------------

Hello, world

dimakuv · 2022-01-12T16:02:55Z

Hmmm, I currently cannot think of anything else to check... Have you tried to build your GSC image with ./gsc build -d (debug mode)? This would give you more Gramine logs.

jpelkonen · 2022-01-12T16:10:59Z

@dimakuv I appreciate your help. I am using the -d mode to build. My next thought is to add more debug statements in the python code to see if I can get more detail.

dimakuv · 2022-05-20T07:47:16Z

@jpelkonen Any updates? Were you able to resolve the issue? If yes, could you share what was wrong?

jpelkonen closed this as completed Jan 7, 2022

jpelkonen reopened this Jan 7, 2022

jogi343 mentioned this issue Apr 19, 2022

gramine-sgx-get-token fails to connect to the AESMD service gramineproject/examples#22

Closed

dimakuv mentioned this issue May 20, 2022

unable to run customized image #58

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gramine-sgx-get-token fails to connect to aesmd from within a container #42

gramine-sgx-get-token fails to connect to aesmd from within a container #42

jpelkonen commented Jan 6, 2022

mkow commented Jan 6, 2022

dimakuv commented Jan 7, 2022

jpelkonen commented Jan 7, 2022

jpelkonen commented Jan 7, 2022

dimakuv commented Jan 7, 2022

jpelkonen commented Jan 7, 2022

dimakuv commented Jan 10, 2022

jpelkonen commented Jan 12, 2022

dimakuv commented Jan 12, 2022

jpelkonen commented Jan 12, 2022

dimakuv commented May 20, 2022

gramine-sgx-get-token fails to connect to aesmd from within a container #42

gramine-sgx-get-token fails to connect to aesmd from within a container #42

Comments

jpelkonen commented Jan 6, 2022

Description of the problem

Steps to reproduce

Expected results

Actual results

Additional information

mkow commented Jan 6, 2022

dimakuv commented Jan 7, 2022

jpelkonen commented Jan 7, 2022

jpelkonen commented Jan 7, 2022

dimakuv commented Jan 7, 2022

jpelkonen commented Jan 7, 2022

dimakuv commented Jan 10, 2022

jpelkonen commented Jan 12, 2022

dimakuv commented Jan 12, 2022

jpelkonen commented Jan 12, 2022

dimakuv commented May 20, 2022