macOS: Support for recent kernels (automagic, modules) #1115

Abyss-W4tcher · 2024-03-23T16:39:24Z

Hello 👋,

This Pull Request aims to refactor the macOS x86/64 automagic layer stacking, as well as support for a recent kernel functionality named "KernelCache".

These changes only add APIs to the framework, without modifying inputs/outputs of existing ones.

Tested on kernels :

10.9.3_build-13D65
10.12.6_build-16G29
10.15.7_build-19H15
12.0.1_build-21A559
13.6.4_build-22G513
14.0_build-23A5257q

Justifications are given, in the form of a comment or a link to the official darwin source code.

Details

KernelCache

The kernelcache is a pre-compiled and pre-linked version of the XNU kernel, with essential device drivers and kernel extensions. It is stored in a compressed format and is decompressed into memory during the boot process. The kernelcache facilitates faster booting by having a ready-to-use version of the kernel and essential drivers available, thereby reducing the time and resources that would otherwise be spent loading and dynamically linking these components at startup.

source (french) : https://book.hacktricks.xyz/v/fr/macos-hardening/macos-security-and-privilege-escalation/mac-os-architecture#kernelcache

This feature impacts symbol resolving, as some need to be shifted with the "classical" KASLR shift, and some with the "vm_kernel_slide" shift. The current shift determining method, via darwin banner scanning, was actually calculating the "vm_kernel_slide" shift, from which it wasn't possible to also determine the KASLR shift. The new implementation takes advantage of the lowGlo kernel structure, which is consistent and available in old and recent kernels, and reveals the KASLR shift value. Banner scanning was also proven to be a bit slower, and more redundant, in its logic.

As discussed in #1114, a new macOS kernel module was added to the framework, which takes care of determining which slide to use to correctly resolve a symbol. Automagic is able to detect and instantiate it automatically, as well.

Additional changes

Some variable names and logics were changed in the mac stacker, to be more straight to the point, while keeping the API compatibility.

If you possess any macOS X samples, please feel free to merge this PR in your local installation, and give feedback !

…he slide to some symbols

ikelos

Thanks very much for this, it looks setup the way I was hoping, but it's also quite involved (not quite as involved as the arm64 layer, but getting close). I've asked @atcuno for a review to make sure the Mac side of things and the slide stuff makes sense. I've given it a once over and it looks like I didn't implement everything needed for Modules properly, so I probably should go fix that first, but this looks very promising. 5:)

ikelos · 2024-03-23T21:15:50Z

volatility3/framework/automagic/mac.py

                if kaslr_shift == 0:
                    vollog.log(
                        constants.LOGLEVEL_VVV,
-                        f"Invalid kalsr_shift found at offset: {banner_offset}",
+                        f"Unable to calculate a valid KASLR shift for banner : {banner}",


Outputting the banner seems less useful than the offset that was being checked to be a kernel? Any reason for the change in debug message?

The banner offset isn't relevant anymore, to detect KASLR. This message can be useful, as there are many different banners scanned in a memory sample. KASLR shift will be banner independant, but the validation stage will stop at version_major and version_minor checks.

It does still get passed to find_aslr though, so either it is still useful, or the signature for find_aslr takes in unnecessary parameters (which should be fixed, but I don't know which version numbers would need bumping to do that safely)...

Exactly, some parameters aren't necessary anymore, but it breaks the API...

volatility3/framework/automagic/mac.py

ikelos · 2024-03-23T21:25:26Z

volatility3/framework/automagic/mac.py

-            version_minor_json_address
-        )
+        banner_major_version, banner_minor_version = [
+            int(x) for x in compare_banner[22:].split(b".")[0:2]


This feels like it will be fragile, could we split this out into a separate function so it's clearer, and maybe hunt for the version number a little better than hard coding 22 bytes in? I realize this was in the previous code, but this gives us an opportunity to tidy it up whilst we're here revamping everything.

I made a change to this logic, by leveraging a regex scanner. If you had something else in mind, do not hesitate to inform me.

ikelos · 2024-03-23T21:29:27Z

volatility3/framework/automagic/mac.py

+                    )
+                )
+
+            if vm_kernel_slide_candidate & 0xFFF != 0:


Should this be the size of a page? If so, this should be the value from the layer.page_size - 1 rather than a hard coded value...

The page size might be architecture independant, even if it's 4kB for macOS x86/64, in the future it might be 16kB or 64kB for AArch64. To determine the page size, we would typically need to read a symbol from the kernel (if it even exists), but is a bit of a "snake biting its tail" situation.

This logic would need to be updated at two other places in this file (dtb aligned and aslr shift aligned).

Well, given it can be different we should at least parameterize these. The page size should be dependent on the architecture, so a value taken from the layer (which is why the layer has the page_size value)...

The LimeLayer, VMWareLayer , FileLayer etc. is embedding the page size ? I don't know how this could be relevant to higher layers (Intel, AArch64), as the page size is typically decided in the kernel config ?

But at least parameterizing it would be more uniform, indeed !

ikelos · 2024-03-23T21:45:59Z

volatility3/framework/contexts/__init__.py

@@ -462,3 +462,149 @@ def __init__(
        Module.__init__(
            self, context, name, layer_name, offset, symbol_table_name, layer_name
        )
+
+
+class MacOSKernelCacheSupportModule(Module):


If you're constructing a new Module with addition configuration items, you'll need to override build_configuration and should define the get_requirements method of the underlying ConfigurableInterface. This has also highlighted that the Module/ModuleInterface classes don't define this (but should do).

It's actually not the Module/ModuleInterface that needs to define this, but the ModuleRequirement, so this might require changing mac plugins to require a mac module? 5:S

volatility3/framework/contexts/__init__.py

ikelos · 2024-03-23T21:57:14Z

volatility3/framework/contexts/__init__.py

+            context.config[pathjoin(config_path, self.layer_name, "kernel_start")]
+            & self.context.layers[self.layer_name].address_mask
+        )
+        context.config[pathjoin(config_path, "kernel_end")] = (


Rather than being a parameter, this should be based off the module offset and size.

In the case of a KernelCache, the offset will be the start of the machO KernelCache __TEXT, and not the "classic" kernel __TEXT start.

Using these parameters allow to use pre-determined values, instead of calculating them "on the fly", risking to mess things up.

FYI, here is a typical configuration, right before exiting mac.py automagic :

{ "kernel_banner": "Darwin Kernel Version 21.1.0: Wed Oct 13 17:33:23 PDT 2021; root:xnu-8019.41.5~1/RELEASE_X86_64\u0000", "kernel_end": 18446743524345970688, "kernel_start": 18446743524335484928, "kernel_virtual_offset": 384188416, "memory_layer": "VmwareLayer", "page_map_offset": 451420160, "vm_kernel_slide": 379650048 }

This is a module, the module takes an offset and a size (or is supposed to). If it needs additional parameters, that's fine, but from the looks of it, it's just used different parameter names for values that actually mean the same as offset and size. The offset is where the module appears in virtual memory, and size is how many bytes it is.

ikelos · 2024-03-23T21:57:46Z

volatility3/framework/contexts/__init__.py

+        context.config[pathjoin(config_path, "vm_kernel_slide")] = context.config[
+            pathjoin(config_path, self.layer_name, "vm_kernel_slide")
+        ]
+        context.config[pathjoin(config_path, "kernel_start")] = (


@#Rather than being a new config option, this should use the existing offset value I believe.

offset (a.k.a kernel_virtual_offset) is different from vm_kernel_slide (in the case of an MH_FILESET KernelCache), and needs to be passed as a config option.

kernel_virtual_offset is what we've typically passed in to the kernel module, but all modules, regardless of which module, have the common values of offset and size, but for the new module you've constructed you've chosen different names. When I add those into the get_requirements of the Module/ModuleInterface you'll see that you're defining the same values but with a different name. I'd prefer we stick with the generic value names that modules already have rather than custom ones for this specific type of module.

Sorry if the following has too much references, but I prefer to justify what I am proposing.

offset in this case is equivalent to vm_kernel_slide, yes, and I can update this.

But offset != kernel_start, as kernel_start is typically stext symbol. Doing so kernel_end - kernel_start is the size of kernel __TEXT, not especially the size of this module, starting from offset offset.

As an example, vm_kernel_slide/offset physical value can be 0x16c10000, but kernel_start is 0x16a10000.

See, as a quick reference (or even here for more details) :

paniclog_append_noflush("Kernel slide: 0x%016lx\n", (unsigned long) vm_kernel_slide); paniclog_append_noflush("Kernel text base: %p\n", (void *)vm_kernel_stext);

Which in the end, is needed to mimic the following behaviour :

/* * Returns true if the address lies in the kernel __TEXT segment range. */ bool kernel_text_contains(vm_offset_t addr) { return vm_kernel_stext <= addr && addr < vm_kernel_etext; }

I consider kernel_start and kernel_end as additional requirements for this specific module.

ikelos · 2024-03-23T21:59:18Z

volatility3/framework/layers/intel.py

@@ -501,3 +501,25 @@ class WindowsIntel32e(WindowsMixin, Intel32e):

    def _translate(self, offset: int) -> Tuple[int, int, str]:
        return self._translate_swap(self, offset, self._bits_per_register // 2)
+
+
+class MacIntelMhFilesetKernelCache(Intel32e):


I'm not sure why these have been added to the layer, rather than the module? I think I probably need to review how configurations for modules are saved.

As discussed on Slack, this is the only reliable way I found to have the config options carried from automagic to the global context config (which are later used in the new MacOSKernelCacheSupportModule.

I think I'm missing something, these should be parameters on the Module and have nothing to do with the layer/architecture? Lemme get the Module/ModuleInterface stuff tidied up and then we can look at what's going on...

Basically, we determine useful values in automagic, and I want them to be accessible when we create the modules in automagic/module.py.

This have nothing to do with the layer indeed, it is just that the stacked layer has a config, and it is saved in the global context, so I can access them later (and it is also saved to the config, which can be exported with --save-config)

I'll wait for your update, I think the modules are missing the get_requirements() as you pointed out, which might help saving the config to the global context ?

ikelos

Thanks for making some of those changes. I'm clearly still quite confused about some of the moving pieces. Specifically why the layer's are getting touched when the only thing that should be impacted are the modules. I'll get a PR out for the changes this has highlighted need making to the Module/ModuleInterface and then we can see how everything fits together after that...

ikelos · 2024-03-24T14:28:58Z

volatility3/framework/automagic/mac.py

+                    )
+                )
+
+            if vm_kernel_slide_candidate & 0xFFF != 0:


Well, given it can be different we should at least parameterize these. The page size should be dependent on the architecture, so a value taken from the layer (which is why the layer has the page_size value)...

ikelos · 2024-03-24T14:33:09Z

volatility3/framework/automagic/mac.py

                if kaslr_shift == 0:
                    vollog.log(
                        constants.LOGLEVEL_VVV,
-                        f"Invalid kalsr_shift found at offset: {banner_offset}",
+                        f"Unable to calculate a valid KASLR shift for banner : {banner}",


It does still get passed to find_aslr though, so either it is still useful, or the signature for find_aslr takes in unnecessary parameters (which should be fixed, but I don't know which version numbers would need bumping to do that safely)...

ikelos · 2024-03-24T14:37:50Z

volatility3/framework/automagic/module.py

+                layer_class_str = context.config[layer_class_config_path]
+                if (
+                    layer_class_str
+                    == "volatility3.framework.layers.intel.MacIntelMhFilesetKernelCache"


Yeah, it should end up as a string in the config, but we want to make sure that string relates to an existing class and that if the file gets moved/renamed etc, an IDE doing refactor can spot the break, hence converting it from an object to a string name. So, yeah, basically convert the python object to the string name (we can make a separate function for clarity) but then compare layer_class_str == class_string_name(class_we_care_about) rather than a fixed string.

You shouldn't ever need to convert from the string to a class?

ikelos · 2024-03-24T14:41:35Z

volatility3/framework/contexts/__init__.py

+        context.config[pathjoin(config_path, "vm_kernel_slide")] = context.config[
+            pathjoin(config_path, self.layer_name, "vm_kernel_slide")
+        ]
+        context.config[pathjoin(config_path, "kernel_start")] = (


kernel_virtual_offset is what we've typically passed in to the kernel module, but all modules, regardless of which module, have the common values of offset and size, but for the new module you've constructed you've chosen different names. When I add those into the get_requirements of the Module/ModuleInterface you'll see that you're defining the same values but with a different name. I'd prefer we stick with the generic value names that modules already have rather than custom ones for this specific type of module.

ikelos · 2024-03-24T14:43:38Z

volatility3/framework/contexts/__init__.py

+            context.config[pathjoin(config_path, self.layer_name, "kernel_start")]
+            & self.context.layers[self.layer_name].address_mask
+        )
+        context.config[pathjoin(config_path, "kernel_end")] = (


This is a module, the module takes an offset and a size (or is supposed to). If it needs additional parameters, that's fine, but from the looks of it, it's just used different parameter names for values that actually mean the same as offset and size. The offset is where the module appears in virtual memory, and size is how many bytes it is.

ikelos · 2024-03-24T14:45:06Z

volatility3/framework/layers/intel.py

@@ -501,3 +501,25 @@ class WindowsIntel32e(WindowsMixin, Intel32e):

    def _translate(self, offset: int) -> Tuple[int, int, str]:
        return self._translate_swap(self, offset, self._bits_per_register // 2)
+
+
+class MacIntelMhFilesetKernelCache(Intel32e):


I think I'm missing something, these should be parameters on the Module and have nothing to do with the layer/architecture? Lemme get the Module/ModuleInterface stuff tidied up and then we can look at what's going on...

ikelos · 2024-03-24T15:19:08Z

Ok, so ModuleRequirements differ from TranslationLayerRequirements and SymbolTableRequirements, in that they didn't have a class and therefore there was only ever expected to be one Module class. Since we're now introducing another potential class, it needs some reworking...

Abyss-W4tcher added 23 commits March 20, 2024 14:18

MH_FILESET kernelcache macOS support

5214601

set join_path and add _KERNEL_MIN_ADDRESS constant

ffcae1c

use _KERNEL_MIN_ADDRESS constant in virtual_to_physical_address

606d6be

remove unnecessary import

4bc21a1

add seen_banners mechanism

2c3db89

more explicit symbol table variable name

6ea2f16

find_aslr caller and logging messages changes

78fc8f7

logging message improvement

3678ac5

use lowGlo scanning for aslr calculation

a8c55ef

vm_kernel_slide calculations

de50148

aslr validation

a01c9c9

kernelcache calculations utilities

83b7d40

save needed constants for later shift use

0d19dbc

add constant config prefix helper for kernelcache

e601ccc

save kernelcache config options in a specific intel layer, if needed

4fd0228

add logging and import path_join

b99c688

introduce kernelcache unslide, to circumvent the additional kernelcac…

e079d99

…he slide to some symbols

revert symbol table sliding, for macOS kernelcache

f6b4996

add kernelcache support module

37f25b5

automagic: use kernelcache module if needed

3e0f975

remove unnecessary macos requirement

e1b7151

revert logging and path_join imports

2e6f675

additional documentation

781d1f0

Abyss-W4tcher mentioned this pull request Mar 23, 2024

macOS : plugins updates, for recent kernels support #1116

Draft

ikelos assigned atcuno Mar 23, 2024

ikelos requested review from atcuno and ikelos March 23, 2024 21:12

ikelos reviewed Mar 23, 2024

View reviewed changes

Abyss-W4tcher added 2 commits March 24, 2024 09:45

remove irrelevant sqlitecache line

09db071

clean comparison

6b57710

Abyss-W4tcher added 4 commits March 24, 2024 10:10

use a regex pattern to determine banner major and minor versions

a5c58a7

lowglo scanner return type hinting

fd37a00

check if offset in kernel boundaries checker is absolute or not

16979f7

remove unnecessary processing in get_symbols_by_absolute_location

bd5f3f3

ikelos reviewed Mar 24, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

macOS: Support for recent kernels (automagic, modules) #1115

macOS: Support for recent kernels (automagic, modules) #1115

Abyss-W4tcher commented Mar 23, 2024

ikelos left a comment

ikelos Mar 23, 2024

Abyss-W4tcher Mar 24, 2024

ikelos Mar 24, 2024

Abyss-W4tcher Mar 24, 2024

ikelos Mar 23, 2024

Abyss-W4tcher Mar 24, 2024

ikelos Mar 23, 2024

Abyss-W4tcher Mar 24, 2024

ikelos Mar 24, 2024

Abyss-W4tcher Mar 24, 2024 •

edited

ikelos Mar 23, 2024

ikelos Apr 23, 2024

ikelos Mar 23, 2024

Abyss-W4tcher Mar 24, 2024

ikelos Mar 24, 2024

ikelos Mar 23, 2024

Abyss-W4tcher Mar 24, 2024

ikelos Mar 24, 2024

Abyss-W4tcher Mar 25, 2024

Abyss-W4tcher Mar 25, 2024

ikelos Mar 23, 2024

Abyss-W4tcher Mar 24, 2024

ikelos Mar 24, 2024

Abyss-W4tcher Mar 24, 2024

ikelos left a comment

ikelos Mar 24, 2024

ikelos Mar 24, 2024

ikelos Mar 24, 2024

ikelos Mar 24, 2024

ikelos Mar 24, 2024

ikelos Mar 24, 2024

ikelos commented Mar 24, 2024

macOS: Support for recent kernels (automagic, modules) #1115

Are you sure you want to change the base?

macOS: Support for recent kernels (automagic, modules) #1115

Conversation

Abyss-W4tcher commented Mar 23, 2024

Details

KernelCache

Additional changes

ikelos left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Abyss-W4tcher Mar 24, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ikelos left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ikelos commented Mar 24, 2024

Abyss-W4tcher Mar 24, 2024 •

edited