A toy x86-64 virtualizing obfuscator

I plan on writing a blog or something about this soon going into more detail. Mostly used to familiarize myself with Rust and determine it's compatibility within this space (it's very good). This project is currently organized as a Rust library but the intention is to just run the test cases at the moment.

This super simple virtualizing obfuscator operates on fully assembled x86-64 byte arrays. You feed it fully assembled x86-64 instructions and it will disassemble them, translate them to a simple stack machine instruction set, and JIT assemble a vmenter and vmexit routine. You can then call the vmenter routine to run the virtualized instructions.

Currently this project only supports virtualization of a very few select x86-64 instructions (namely the ones specifically required by the function I chose to target), but adding support for more is easy. This could easily be used as the starting point for a more fully featured virtualizing obfuscation system.

The instructions I chose were for the default godbolt function compiled by MSVC and GCC:

// Type your code here, or load an example.
int square(int num) {
    return num * num;
}

MSVC:

mov     DWORD PTR [rsp+8], ecx
mov     eax, DWORD PTR num$[rsp]
imul    eax, DWORD PTR num$[rsp]
ret     0

GCC:

push    rbp
mov     rbp, rsp
mov     DWORD PTR [rbp-4], edi
mov     eax, DWORD PTR [rbp-4]
imul    eax, eax
pop     rbp
ret