clock_gettime returning 4ms increments

[milan-enclave]

@thomasten calling time.Now() (clock_gettime under the hood) and compiling it program with ego:

package main

import (
	"fmt"
	"time"
)

func main() {
	last := int64(0)
	start := time.Now()
	for i := 0; i < 100000; i++ {
		now := time.Now().UnixNano() / 1e6
		if last != now {
			last = now
			fmt.Println(now)
		}
	}
	fmt.Println(time.Now().Sub(start))
}

When running inside the enclave vs. outside of it, I am getting 4ms gaps between time measurements. Also the loop itself takes a lot longer in enclave:

~/ego/samples/clock$ ego run clock
EGo v1.4.1 (009751335951262b22b514412c7b45e33705681b)
[erthost] loading enclave ...
[erthost] entering enclave ...
[ego] starting application ...
1703239563096
1703239563100
1703239563104
1703239563108
11.99988ms

~/ego/samples/clock$ ./clock
1703239569361
1703239569362
1703239569363
1703239569364
1703239569365
3.780436ms

I can see that you have patched out vdso implementation of clock_gettime inside the ertgo - and also disable vdso in the openenclave patch - and instead go for the syscall - the whole point of vdso is so that clock_gettime calls are faster, as the data structures are updated by kernel itself, and therefore syscall doesn't need to happen. I can also see removal of rdtsc calls - these I understand are not supported on production SGX.

I wonder why remove vdso support and go for slower syscall instead?

Regardless, this alone doesnt explain why there are 4ms between clock_gettime calls, as if I run outside of the enclave I can see 1ms increments. I have stepped through the time.now() call via ego-gdb (and having the signed binary be configured with debug: true), and I can see that the patched syscall ends up in custom libc implementation, that does load vdso entry directly.

Is this something that Intel SGX does to prevent timing attacks? Is there a way to get actual accurate time inside the enclave?

[thomasten]

Hi,

The vdso implementation inside ertgo is patched out because the vdso pointers would need to be initialized to point to some emulation code inside the enclave. We found it easier to replace the call with a call to (fast emulated) clock_gettime instead of additionally implementing such code.

vdso in Open Enclave is not used for time, but only as a way to handle exceptions. When this feature was added, we experienced system freezes in some high-load situations and thus disabled it. The alternative exception handling, which previously had been the standard method, has since been sufficient for EGo.

You already found out that our clock_gettime implementation loads the vdso entry directly and doesn't do an actual syscall. I assume you stepped into https://github.com/edgelesssys/edgelessrt/blob/5365e10d10ff4ec108cb278b566e60222f812917/src/ertlibc/time.cpp#L97

Here you can also see that the implementation only supports coarse time because of the missing rdtsc. This is the reason for the 4ms increments.

On Icelake and newer, SGX actually supports rdtsc. It may be possible to extend the implementation to provide fine-grained time. If you have the required hardware and are able to implement this, we would welcome a contribution.

Regarding the loop taking longer when running inside the enclave, the (relative) difference gets much smaller on my machine if I set the loop count, e.g., 100 times higher.

[milan-enclave]

Thank you for explanation. Would you agree that this would fix it:

• Fill in vDSO entry point for CLOCK_REALTIME (or also CLOCK_MONOTONIC) here: https://github.com/edgelesssys/edgelessrt/blob/5365e10d10ff4ec108cb278b566e60222f812917/src/ert/host/vdso.cpp#L20

• Then make sure that the https://github.com/edgelesssys/edgelessrt/blob/5365e10d10ff4ec108cb278b566e60222f812917/src/ertlibc/time.cpp#L97 actually sets the proper clock id routines in vDSO. That should be it.

• Update EDL function signature: https://github.com/edgelesssys/edgelessrt/blob/5365e10d10ff4ec108cb278b566e60222f812917/src/ert/include/openenclave/edl/ertlibc.edl#L19

• Detect that SGX 2 is supported:

RDTSC and RDTSCP are legal inside an enclave for processors that support SGX2 (subject to the value of CR4.TSD). For processors which support SGX1 but not SGX2, RDTSC and RDTSCP will cause #UD.

https://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-software-developer-vol-3d-part-4-manual.pdf

[thomasten]

These changes are required, yes, but there's more. The pointers are not the vDSO routines, but the memory where the kernel writes the timestamps. These are used for reading the time here:
https://github.com/edgelesssys/edgelessrt/blob/5365e10d10ff4ec108cb278b566e60222f812917/src/ertlibc/time.cpp#L139-L150
So this is a replacement for what usually would be done in linux-vdso.so, which EdgelessRT can't load into the enclave.

If we want high-res time, the timestamp would need to be adjusted by the value obtained by RDTSC. I think the corresponding Linux code is this:
https://github.com/torvalds/linux/blob/808094fcbf4196be0feb17afbbdc182ec95c8cec/lib/vdso/gettimeofday.c#L121-L152
I don't know how exactly this works.

So in addition to your suggested changes:

• Understand how the timestamp is adjusted by the CPU's TSC. Add this to sc::clock_gettime as a clean-room reimplementation to not violate Linux's GPL license. (E.g., read the Linux code only to determine the behavior regarding the timestamp adjustment. Then implement it based only on the learned behavior.)