Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compiling on macOS fails #3

Open
dcow opened this issue Jul 28, 2017 · 15 comments
Open

Compiling on macOS fails #3

dcow opened this issue Jul 28, 2017 · 15 comments

Comments

@dcow
Copy link

dcow commented Jul 28, 2017

On macOS make yeilds:

/usr/include/ucontext.h:43:2: error: The deprecated ucontext routines require _XOPEN_SOURCE to be defined
#error The deprecated ucontext routines require _XOPEN_SOURCE to be defined
 ^
...

Using

cc -D_XOPEN_SOURCE -c injector.c -o injector.o -Wall

or

diff --git a/injector.c b/injector.c
index 75848b5..280ae4e 100644
--- a/injector.c
+++ b/injector.c
@@ -13,7 +13,7 @@
 #include <time.h>
 #include <execinfo.h>
 #include <limits.h>
-#include <ucontext.h>
+#include <sys/ucontext.h>
 #include <sys/types.h>
 #include <stdint.h>
 #include <stdbool.h>

fixes the first issue. But, it looks like there is still an issue with the portability of ucontext structures:

cc  -c injector.c -o injector.o -Wall
injector.c:321:93: warning: excess elements in array initializer
        .start={.bytes={0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00}, .len=0},
                                                                                                   ^~~~
injector.c:322:91: warning: excess elements in array initializer
        .end={.bytes={0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff}, .len=0},
                                                                                                 ^~~~
injector.c:853:31: error: member reference type 'struct __darwin_mcontext64 *' is a pointer; did you mean to use '->'?
        ((ucontext_t*)p)->uc_mcontext.gregs[IP]+=UD2_SIZE;
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
                                     ->
injector.c:853:32: error: no member named 'gregs' in 'struct __darwin_mcontext64'
        ((ucontext_t*)p)->uc_mcontext.gregs[IP]+=UD2_SIZE;
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^
injector.c:853:38: error: use of undeclared identifier 'REG_RIP'
        ((ucontext_t*)p)->uc_mcontext.gregs[IP]+=UD2_SIZE;
                                            ^
injector.c:81:13: note: expanded from macro 'IP'
        #define IP REG_RIP
                   ^
injector.c:866:29: error: member reference type 'struct __darwin_mcontext64 *' is a pointer; did you mean to use '->'?
                (uintptr_t)uc->uc_mcontext.gregs[IP]-(uintptr_t)packet-preamble_length;
                           ~~~~~~~~~~~~~~~^
                                          ->
injector.c:866:30: error: no member named 'gregs' in 'struct __darwin_mcontext64'
                (uintptr_t)uc->uc_mcontext.gregs[IP]-(uintptr_t)packet-preamble_length;
                           ~~~~~~~~~~~~~~~ ^
injector.c:866:36: error: use of undeclared identifier 'REG_RIP'
                (uintptr_t)uc->uc_mcontext.gregs[IP]-(uintptr_t)packet-preamble_length;
                                                 ^
injector.c:81:13: note: expanded from macro 'IP'
        #define IP REG_RIP
                   ^
injector.c:883:24: error: member reference type 'struct __darwin_mcontext64 *' is a pointer; did you mean to use '->'?
        memcpy(uc->uc_mcontext.gregs, fault_context.gregs, sizeof(fault_context.gregs));
               ~~~~~~~~~~~~~~~^
                              ->
/usr/include/secure/_string.h:65:27: note: expanded from macro 'memcpy'
  __builtin___memcpy_chk (dest, src, len, __darwin_obsz0 (dest))
                          ^~~~
injector.c:883:25: error: no member named 'gregs' in 'struct __darwin_mcontext64'
        memcpy(uc->uc_mcontext.gregs, fault_context.gregs, sizeof(fault_context.gregs));
               ~~~~~~~~~~~~~~~ ^
/usr/include/secure/_string.h:65:27: note: expanded from macro 'memcpy'
  __builtin___memcpy_chk (dest, src, len, __darwin_obsz0 (dest))
                          ^~~~
injector.c:883:45: error: member reference type 'mcontext_t' (aka 'struct __darwin_mcontext64 *') is a pointer; did you mean to use
      '->'?
        memcpy(uc->uc_mcontext.gregs, fault_context.gregs, sizeof(fault_context.gregs));
                                      ~~~~~~~~~~~~~^
                                                   ->
/usr/include/secure/_string.h:65:33: note: expanded from macro 'memcpy'
  __builtin___memcpy_chk (dest, src, len, __darwin_obsz0 (dest))
                                ^~~
injector.c:883:46: error: no member named 'gregs' in 'struct __darwin_mcontext64'
        memcpy(uc->uc_mcontext.gregs, fault_context.gregs, sizeof(fault_context.gregs));
                                      ~~~~~~~~~~~~~ ^
/usr/include/secure/_string.h:65:33: note: expanded from macro 'memcpy'
  __builtin___memcpy_chk (dest, src, len, __darwin_obsz0 (dest))
                                ^~~
injector.c:883:73: error: member reference type 'mcontext_t' (aka 'struct __darwin_mcontext64 *') is a pointer; did you mean to use
      '->'?
        memcpy(uc->uc_mcontext.gregs, fault_context.gregs, sizeof(fault_context.gregs));
                                                                  ~~~~~~~~~~~~~^
                                                                               ->
/usr/include/secure/_string.h:65:38: note: expanded from macro 'memcpy'
  __builtin___memcpy_chk (dest, src, len, __darwin_obsz0 (dest))
                                     ^~~
injector.c:883:74: error: no member named 'gregs' in 'struct __darwin_mcontext64'
        memcpy(uc->uc_mcontext.gregs, fault_context.gregs, sizeof(fault_context.gregs));
                                                                  ~~~~~~~~~~~~~ ^
/usr/include/secure/_string.h:65:38: note: expanded from macro 'memcpy'
  __builtin___memcpy_chk (dest, src, len, __darwin_obsz0 (dest))
                                     ^~~
injector.c:883:24: error: member reference type 'struct __darwin_mcontext64 *' is a pointer; did you mean to use '->'?
        memcpy(uc->uc_mcontext.gregs, fault_context.gregs, sizeof(fault_context.gregs));
               ~~~~~~~~~~~~~~~^
                              ->
/usr/include/secure/_string.h:65:59: note: expanded from macro 'memcpy'
  __builtin___memcpy_chk (dest, src, len, __darwin_obsz0 (dest))
                                                          ^~~~
/usr/include/secure/_common.h:38:55: note: expanded from macro '__darwin_obsz0'
#define __darwin_obsz0(object) __builtin_object_size (object, 0)
                                                      ^~~~~~
injector.c:883:25: error: no member named 'gregs' in 'struct __darwin_mcontext64'
        memcpy(uc->uc_mcontext.gregs, fault_context.gregs, sizeof(fault_context.gregs));
               ~~~~~~~~~~~~~~~ ^
/usr/include/secure/_string.h:65:59: note: expanded from macro 'memcpy'
  __builtin___memcpy_chk (dest, src, len, __darwin_obsz0 (dest))
                                                          ^~~~
/usr/include/secure/_common.h:38:55: note: expanded from macro '__darwin_obsz0'
#define __darwin_obsz0(object) __builtin_object_size (object, 0)
                                                      ^~~~~~
injector.c:884:17: error: member reference type 'struct __darwin_mcontext64 *' is a pointer; did you mean to use '->'?
        uc->uc_mcontext.gregs[IP]=(uintptr_t)&resume;
        ~~~~~~~~~~~~~~~^
                       ->
injector.c:884:18: error: no member named 'gregs' in 'struct __darwin_mcontext64'
        uc->uc_mcontext.gregs[IP]=(uintptr_t)&resume;
        ~~~~~~~~~~~~~~~ ^
injector.c:884:24: error: use of undeclared identifier 'REG_RIP'
        uc->uc_mcontext.gregs[IP]=(uintptr_t)&resume;
                              ^
injector.c:81:13: note: expanded from macro 'IP'
        #define IP REG_RIP
                   ^
injector.c:885:17: error: member reference type 'struct __darwin_mcontext64 *' is a pointer; did you mean to use '->'?
        uc->uc_mcontext.gregs[REG_EFL]&=~TF;
        ~~~~~~~~~~~~~~~^
                       ->
injector.c:885:18: error: no member named 'gregs' in 'struct __darwin_mcontext64'
        uc->uc_mcontext.gregs[REG_EFL]&=~TF;
        ~~~~~~~~~~~~~~~ ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
2 warnings and 20 errors generated.
make: *** [injector.o] Error 1
@orf
Copy link

orf commented Jul 28, 2017

Same, I've tried with the bundled MacOS llvm as well as the latest.

@floatingatoll
Copy link

Pull #6 fixes one of these errors.

Changing <ucontext.h> to <sys/ucontext.h> fixes another.

Changing uc->mcontext.gregs to uc->mcontext->gregs fixed another.

Unfortunately, there's no ->gregs member in uc_mcontext64 on Darwin:

  _STRUCT_MCONTEXT64
  {
          _STRUCT_X86_EXCEPTION_STATE64   __es;
          _STRUCT_X86_THREAD_STATE64      __ss;
          _STRUCT_X86_FLOAT_STATE64       __fs;
  };

Because, apparently, this is dependent on glibc's register reading stuff:

https://fossies.org/dox/glibc-2.25/structmcontext__t.html

This ancient article talks about trying to persuade an old OS X kernel to divulge register state:

http://web.archive.org/web/20090627062246/http://www.matasano.com/log/1100/what-ive-been-doing-on-my-summer-vacation-or-it-has-to-work-otherwise-gdb-wouldnt/

Ideally there's a better way these days, but no guarantee there.

@floatingatoll
Copy link

Happily, ->mcontext64->__ss is nothing but registers:

  #if __DARWIN_UNIX03
  #define _STRUCT_X86_THREAD_STATE64      struct __darwin_x86_thread_state64
  _STRUCT_X86_THREAD_STATE64
  {
          __uint64_t      __rax;
          __uint64_t      __rbx;
          __uint64_t      __rcx;

@floatingatoll
Copy link

-       ((ucontext_t*)p)->uc_mcontext.gregs[IP]+=UD2_SIZE;
+       ((ucontext_t*)p)->uc_mcontext->__ss.__rip+=UD2_SIZE;

Variations of this seem to, at the very least, remove more warnings. Still doesn't compile and I'd be surprised if a straight swap out here worked.

@floatingatoll
Copy link

cpu_set_t is GNU-only:

https://www.gnu.org/software/libc/manual/html_node/CPU-Affinity.html

You can fake the goal here, which is I believe to test one core only, using:

http://jesperrasmussen.com/2013/03/07/limiting-cpu-cores-on-the-fly-in-os-x/

Storing and restoring __ss might be as simple as using plain struct assignment:

-       memcpy(uc->uc_mcontext->gregs, fault_context->gregs, sizeof(fault_context->gregs));
+        uc->uc_mcontext->__ss = fault_context->__ss;

And then there's this:

injector.c:778:24: error: invalid operand for inline asm constraint 'i'
        __asm__ __volatile__ ("\

@floatingatoll
Copy link

Which I tried to resolve with 'static dummy', and then managed to get:

cc -c injector.c -o injector.o -Wall
fatal error: error in backend: 32-bit absolute addressing is not supported in 64-bit mode

So we're making progress! But this is where I run out of gas, because I can't debug assembler 32/64 differences, sorry.

https://stackoverflow.com/questions/35834937/error-in-backend-32-bit-absolute-addressing-is-not-supported-in-64-bit-mode

@IComplainInComments
Copy link

IComplainInComments commented Aug 3, 2017

So it seems the issues we are facing for macOS deal with the XNU kernel architecture as a whole.

  1. Instead of using gLibC, we need to use the built-in XNU kernel PTrace calls.

  2. We need to modify CPU_SET to use Apples alternative. Which can be described here: http://yyshen.github.io/2015/01/18/binding_threads_to_cores_osx.html.

So in general this code will work, but will need a specific branch for any MACH based micro kernel.

@IComplainInComments
Copy link

IComplainInComments commented Aug 3, 2017

So I am working on CPU_SET, and it seems all the functions needed can be called from the Affinity API (According to http://yyshen.github.io/2015/01/18/binding_threads_to_cores_osx.html)

So far the first code block works fine, and compiles with no errors. Also with these code blocks in, the current CPU_SET errors go away, but... this code block needs rewritten it seems.

`int pthread_setaffinity_np(pthread_t thread, size_t cpu_size,
cpu_set_t *cpu_set)
{
pthread_cond_t mach_thread;
int core = 0;

for (core = 0; core < 8 * cpu_size; core++) {
    if (CPU_ISSET(core, cpu_set)) break;
}
printf("binding to core %d\n", core);
thread_affinity_policy_data_t policy = { core };
mach_thread = pthread_mach_thread_np(thread);
thread_policy_set(mach_thread, THREAD_AFFINITY_POLICY,
                  (thread_policy_set)&policy, 1);
return 0;

}`

@floatingatoll Would you be able to debug this code? I unfortunately do not know enough C to try.

These are the errors I get:

injector.c:1383:5: error: use of undeclared identifier 'thread_affinity_policy_data_t' thread_affinity_policy_data_t policy = { core }; ^ injector.c:1384:17: error: assigning to 'pthread_cond_t' (aka 'struct _opaque_pthread_cond_t') from incompatible type 'mach_port_t' (aka 'unsigned int') mach_thread = pthread_mach_thread_np(thread); ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ injector.c:1385:5: warning: implicit declaration of function 'thread_policy_set' is invalid in C99 [-Wimplicit-function-declaration] thread_policy_set(mach_thread, THREAD_AFFINITY_POLICY, ^ injector.c:1386:43: error: use of undeclared identifier 'policy' (thread_policy_set)&policy, 1); ^ injector.c:1385:36: error: use of undeclared identifier 'THREAD_AFFINITY_POLICY' thread_policy_set(mach_thread, THREAD_AFFINITY_POLICY, ^ injector.c:1396:7: warning: implicit declaration of function 'sched_setaffinity' is invalid in C99 [-Wimplicit-function-declaration] if (sched_setaffinity(0, sizeof(mask), &mask)) {
EDIT: The Affinity API documentation is here: https://developer.apple.com/library/content/releasenotes/Performance/RN-AffinityAPI/index.html

It says Leopard, but it is still supported and used in the current macOS (Including 10.13)

@floatingatoll
Copy link

floatingatoll commented Aug 3, 2017 via email

@IComplainInComments
Copy link

Alright it seems like adding #include <mach/thread_policy.h> does indeed do as you said @floatingatoll

There are now just 8 total errors that need hammered out.

The new Errors go as follows:
injector.c:1385:17: error: assigning to 'pthread_cond_t' (aka 'struct _opaque_pthread_cond_t') from incompatible type 'mach_port_t' (aka 'unsigned int') mach_thread = pthread_mach_thread_np(thread);

and

injector.c:1387:42: error: invalid operands to binary expression ('int (*)()' and 'thread_affinity_policy_data_t' (aka 'struct thread_affinity_policy')) (thread_policy_set)&policy, 1); ~~~~~~~~~~~~~~~~~~~^~~~~~~

@IComplainInComments
Copy link

Found a fix, delete this code block:

`
int pthread_setaffinity_np(pthread_t thread, size_t cpu_size,
cpu_set_t *cpu_set)
{
pthread_cond_t mach_thread;
int core = 0;

for (core = 0; core < 8 * cpu_size; core++) {
    if (CPU_ISSET(core, cpu_set)) break;
}
printf("binding to core %d\n", core);
thread_affinity_policy_data_t policy = { core };
mach_thread = pthread_mach_thread_np(thread);

thread_policy_set(mach_thread, THREAD_AFFINITY_POLICY,
(thread_policy_set)&policy, 1);
return 0;
}`

6 Errors left! So all we need to do now is figuring out how to get the Program to use Pthread from the kernel!

@IComplainInComments
Copy link

Also stuck at injector.c:818:24: error: invalid operand for inline asm constraint 'i' __asm__ __volatile__ ("\ ^

:/

@floatingatoll
Copy link

floatingatoll commented Aug 3, 2017 via email

@jaketesler
Copy link

I got this to compile, but I'm not sure if the executable still works as intended.
This is ONLY for 64-bit – I didn't take the time to convert these changes for 32-bit, but I imagine it wouldn't be too difficult.


Change 1: Dummy Struct

Modify the following snippet:

sandsifter/injector.c

Lines 174 to 177 in 8375e61

struct {
uint64_t dummy_stack_hi[256];
uint64_t dummy_stack_lo[256];
} dummy_stack __attribute__ ((aligned(PAGE_SIZE)));

Delete the entire dummy_stack definition. Then, copy the following (to anywhere before the inject_state typedef struct definition on line 90!):

typedef struct {
	uint64_t dummy_stack_hi[256];
	uint64_t dummy_stack_lo[256];
} dumb_stack;
static dumb_stack dummy_stack __attribute__ ((aligned(PAGE_SIZE)));

Then, in the __x86_64__ inject_state definition, modify the struct:

...
	uint64_t rsp;
	dumb_stack dumb;
} state_t;
...

and the inject_state definition:

...
	.rsp=0,
	.dumb.dummy_stack_lo=0,
	.dumb.dummy_stack_hi=0
};
...

We also need to modify the stack reset on line 766:

inject_state.dumb.dummy_stack_lo[0]=0;

And lastly, we need to modify the inline asm here:

sandsifter/injector.c

Lines 777 to 815 in 8375e61

#if __x86_64__
__asm__ __volatile__ ("\
mov %[rax], %%rax \n\
mov %[rbx], %%rbx \n\
mov %[rcx], %%rcx \n\
mov %[rdx], %%rdx \n\
mov %[rsi], %%rsi \n\
mov %[rdi], %%rdi \n\
mov %[r8], %%r8 \n\
mov %[r9], %%r9 \n\
mov %[r10], %%r10 \n\
mov %[r11], %%r11 \n\
mov %[r12], %%r12 \n\
mov %[r13], %%r13 \n\
mov %[r14], %%r14 \n\
mov %[r15], %%r15 \n\
mov %[rbp], %%rbp \n\
mov %[rsp], %%rsp \n\
jmp *%[packet] \n\
"
: /* no output */
: [rax]"m"(inject_state.rax),
[rbx]"m"(inject_state.rbx),
[rcx]"m"(inject_state.rcx),
[rdx]"m"(inject_state.rdx),
[rsi]"m"(inject_state.rsi),
[rdi]"m"(inject_state.rdi),
[r8]"m"(inject_state.r8),
[r9]"m"(inject_state.r9),
[r10]"m"(inject_state.r10),
[r11]"m"(inject_state.r11),
[r12]"m"(inject_state.r12),
[r13]"m"(inject_state.r13),
[r14]"m"(inject_state.r14),
[r15]"m"(inject_state.r15),
[rbp]"m"(inject_state.rbp),
[rsp]"i"(&dummy_stack.dummy_stack_lo),
[packet]"m"(packet)
);

The asm on line 813 should read:

[rsp]"m"(inject_state.dumb.dummy_stack_lo),

Change 2: uc_mcontext

We need to modify uc_mcontext references for macOS. Make all of the following changes:

  • Line 853:
((ucontext_t*)p)->uc_mcontext->__ss.__rip+=UD2_SIZE;
  • Line 866:
(uintptr_t)uc->uc_mcontext->__ss.__rip-(uintptr_t)packet-preamble_length;
  • Lines 883-885:
uc->uc_mcontext->__ss = fault_context->__ss;
uc->uc_mcontext->__ss.__rip =(uintptr_t)&resume;
uc->uc_mcontext->__ss.__rflags &=~TF;

Change 3: header file

Create a new file injector.h with the following contents:

extern char debug, resume, preamble_start, preamble_end;

Then modify the extern definition on line 366.

extern char debug, resume, preamble_start, preamble_end;

Change line 366 to:

char debug, resume, preamble_start, preamble_end;

Change 4: pthread modifications

This code was borrowed from https://yyshen.github.io/2015/01/18/binding_threads_to_cores_osx.html.
Copy the following block underneath the top #include section.

#define SYSCTL_CORE_COUNT   "machdep.cpu.core_count"

typedef struct cpu_set {
  uint32_t    count;
} cpu_set_t;

static inline void
CPU_ZERO(cpu_set_t *cs) { cs->count = 0; }

static inline void
CPU_SET(int num, cpu_set_t *cs) { cs->count |= (1 << num); }

static inline int
CPU_ISSET(int num, cpu_set_t *cs) { return (cs->count & (1 << num)); }

int sched_getaffinity(pid_t pid, size_t cpu_size, cpu_set_t *cpu_set)
{
  int32_t core_count = 0;
  size_t  len = sizeof(core_count);
  int ret = sysctlbyname(SYSCTL_CORE_COUNT, &core_count, &len, 0, 0);
  if (ret) {
    printf("error while get core count %d\n", ret);
    return -1;
  }
  cpu_set->count = 0;
  for (int i = 0; i < core_count; i++) {
    cpu_set->count |= (1 << i);
  }

  return 0;
}

int pthread_setaffinity_np(pthread_t thread, size_t cpu_size,
                           cpu_set_t *cpu_set)
{
  thread_port_t mach_thread;
  int core = 0;

  for (core = 0; core < 8 * cpu_size; core++) {
    if (CPU_ISSET(core, cpu_set)) break;
  }
  printf("binding to core %d\n", core);
  thread_affinity_policy_data_t policy = { core };
  mach_thread = pthread_mach_thread_np(thread);
  thread_policy_set(mach_thread, THREAD_AFFINITY_POLICY,
                    (thread_policy_t)&policy, 1);
  return 0;
}

Then, modify line 1344 to read:

if (pthread_setaffinity_np(0, sizeof(mask), &mask)) {

Lastly, append this to the #include section

#include <mach/thread_policy.h>

Change 5: Python

The sifter.py file is designed for Linux and attempts to gather CPU info via /proc/cpuinfo, which doesn't exist on macOS. So, I made a small change to allow running on a Mac.

sandsifter/sifter.py

Lines 681 to 684 in 8375e61

def get_cpu_info():
with open("/proc/cpuinfo", "r") as f:
cpu = [l.strip() for l in f.readlines()[:7]]
return cpu

Modify the get_cpu_info() function to:

def get_cpu_info():
    p =  subprocess.Popen(['/usr/sbin/sysctl','-a'], stdout=subprocess.PIPE)
    p.wait()
    cpu = [l.strip('machdep.cpu.') for l in str(p.stdout.read()).split('\n') if 'machdep.cpu' in l]
    return cpu

Lastly, run make. If you encounter this error:

/usr/include/ucontext.h:43:2: error: The deprecated ucontext routines require _XOPEN_SOURCE to be defined

Run make with the CFLAGS variable:

CFLAGS="-D_XOPEN_SOURCE" make

Hope this helps! Also hopefully the injector binary still functions as expected. Please comment below with results or findings.

@IComplainInComments
Copy link

@jaketesler Thanks for writing the missing C code we needed! I really should brush up on my coding hehe

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants