Unsupported architecture Apple Silicon M1

[miklaine]

Would it be possible to add such support?

[steven-michaud]

I don't know whether or not it will be possible, but I definitely will look into it. And not just for the M1 chip.

However, I won't start until Apple has made available the source code for the macOS 11 xnu kernel. And I'm inclined to wait to buy an Apple Silicon machine until a better chip than the M1 is available -- until the machines support more than 16GB of RAM.

[steven-michaud]

Apple has recently released source code for the macOS 11.0.1 xnu kernel. But release versions of VMWare Fusion and Parallels don't yet support Apple Silicon hosts and guests. I'm going to wait until at least one of these is available, in a release version, with comprehensive support for Apple Silicon. Adding Apple Silicon support to HookCase, or even just investigating if it's possible to add support, will trigger hundreds of kernel panics in my testing environment. I don't feel comfortable doing this on actual hardware.

[miklaine]

I understand that. Hopefully, Parallels and/or VMWare will release working solutions sooner than later :)

[panjiancai]

I am looking forward to supporting M1 chips.

[steven-michaud]

And I'm waiting for VMware or Parallels to release software that can virtualize some version of macOS on Apple Silicon hardware. As best I can tell, this isn't going to happen soon. Parallels has a beta that can do this in a very limited fashion. VMware doesn't have anything at all. Neither company talks about their future plans, beyond vaguely promising it at some future date.

Even when one or more of these becomes available, I'll have at least several months of very difficult work, without any guarantee that it will be successful.

Given what I know now (which admittedly isn't much), I'd say that a version of HookCase that supports Apple Silicon hardware is at least a year away.

[sferrini]

Hey @steven-michaud, fyi there are two open-source projects on GitHub that use the Virtualization.framework APIs to virtualize macOS on Apple Silicon. One of them (VirtualApple) even exposes a GDB stub port where one can attach lldb and perform kernel debugging.

• https://github.com/KhaosT/MacVM.git
• https://github.com/saagarjha/VirtualApple

I've personally tried both of them, and I was able to successfully boot an Apple Silicon VM.

[steven-michaud]

Thanks, @sferrini, for pointing these out to me. I'll play around with them later, when I have more time.

Neither of them has any documentation, so it's hard to know whether or not they'll meet my needs. These are, briefly: The VMs they create must 1) allow me to run Apple's installer for (say) macOS Monterey (for Apple Silicon) inside them, then 2) on reboot allow me to run macOS Monterey itself (the copy I've just installed). They must also 3) have full access (from inside the VM) to the host machine's hardware -- to things like a dedicated hard disk partition and to everything needed for networking (something like VMware's "bridged networking"). It'd also be nice to 4) be able to copy and paste between the VM and the host machine, but that's not strictly necessary.

I'd like to learn more about Apple's Virtualization framework. I notice Apple has its own documentation. But, given past experience with Apple's docs, this probably isn't enough by itself to allow me to do my own virtualization. It'd probably help to have the example code from these projects.

In any case, learning Apple's Virtualization framework will almost certainly be very time consuming. Add that to the time I already expect to be spending adding support for Apple Silicon to HookCase :-)

[cakarlen]

Has there been any movement on this issue?

[steven-michaud]

No. VMware and Parallels still don't have releases that support macOS clients on Apple Silicon hardware.

If I had to guess, I'd say this is probably because Apple's Virtualization framework is still inadequate for full-featured macOS clients on Apple Silicon hardware. With luck, Apple may provide better support in the next version of macOS ... or maybe they won't. Only time will tell.

[steven-michaud]

Here's a link to a comment by a VMware employee that sheds some light on the problem of supporting macOS clients on Apple Silicon hardware. Note that it's a bit old (11-30-2021). If I find more interesting links I'll post them here.

https://communities.vmware.com/t5/Fusion-for-Apple-Silicon-Tech/Can-you-share-any-info-about-macOS-guest-support/m-p/2880768#M400

[cakarlen]

Would this utility help create VMs on M1? https://github.com/saagarjha/VirtualApple

[steven-michaud]

It might, I suppose. But like the two projects mentioned above, it has no documentation whatsoever.

When I use unknown software, I like to have some idea of what it does before I use it. Especially if it purports to do something as complex as creating virtual machines. Quite frankly, I'm not going to play with these projects until I understand them much better. Practically speaking, the only way for me to do that is to read Apple's Virtualization framework docs and try to write my own virtualization software, using code from these projects as sample code.

As I mentioned above, I now strongly suspect that Apple's Virtualization framework is inadequate for full-featured macOS clients on Apple Silicon hardware. So I'm not going to start this work until the next major release of macOS, and until Apple has had time to rewrite its docs to reflect any changes it might make to Apple's resources for virtualization on Apple Silicon hardware. I'll also wait as long as I can for VMware and Parallels to do this work for me. No sense in doing more than I really need to.

[sferrini]

@steven-michaud if you have some time, you could have a look at the UTM project, they added support for macOS guests in v3.0.0 (they are at v3.1.5 now) using Apple's Virtualization.framework as a backend.

Creating ARM64e macOS VMs with UTM is extremely easy and intuitive, they also integrated the IPSW downloading feature in their UI.

Give it a try, and let me know if you have any issues or questions.
I'm not a maintainer but I've been using this tool successfully in the past few months.

You can download latest installer from here: https://mac.getutm.app/ (or directly from GitHub)

[steven-michaud]

Thanks, @sferrini. I looked briefly at the UTM project, and it does seem promising. I expect, though, that it's different from VMware Fusion and Parallels in (at least) two important ways:

1. VMware Fusion and Parallels do virtualization. But UTM seems to do virtualization on top of emulation (provided via QEMU). This is potentially much more powerful. For example, in principle you'd be able to run a macOS ARM64E client on X86_64 hardware, or in a Windows or Linux host. But it'd be much more resource intensive. And you'd have to emulate every piece of hardware the client needs -- not just pass through access to it (as Fusion and Parallels seem to do).

2. As I understand it, VMware and Parallels work closely with Apple to make their macOS client environments resemble "genuine" Apple hardware as much as possible. They make this easier by forcing their macOS hosts to run on Apple hardware. But I expect UTM's maintainers work entirely independently from Apple (as I do). So running macOS in their virtualized hardware environment is probably a bit like running macOS on a Hackintosh.

At some point I'll try out UTM. I don't expect it to be a straight-up replacement for something that VMware or Parallels might provide in the future. But it may be good enough for me to start work there on porting HookCase to Apple Silicon.

It will be quite a while before I start this work. I'd still much prefer to wait for Apple Silicon macOS clients from VMware or Parallels. But I won't wait forever.

[steven-michaud]

Digging a bit further into UTM, I found this:

"Run multiple instances of macOS on your Apple Silicon Mac with UTM. This can be useful for developers as well as security conscious users. Note that macOS VM support is limited to ARM based Macs running macOS Monterey or higher."

It sounds like this "virtualization" is (in effect) running copies of the host version of macOS in separate windows. Each of these windows is (in effect) chrooted. Unlike with VMware Fusion and Parallels, you can't run different versions of macOS in these "clients" (aka windows). Am I right?

And another question: If you trigger a kernel panic in one of these windows, will it just effect the window? Or will the "host" also panic? If so, this functionality is of no use to me. I need macOS clients that (like VMware clients) can panic without taking down the host.

[sferrini]

You can take a look at UTM's architecture, where they show the following graph:

┌────────────────────┬──────────────────────┐
│   iOS VM Display   │   macOS VM Display   │
├────────────┬───────┴──────────────────────┤
│ iOS Legacy │           SwiftUI            │
├────────────┴──────────────────────────────┤
│             UTMVirtualMachine             │
├────────────────┬──────────────────────────┤
│   CocoaSpice   │                          │
├────────────────┤ Virtualization.framework │
│ QEMU (TCG/HVF) │                          │
└────────────────┴──────────────────────────┘

UTM abstracts the backend with UTMVirtualMachine, and depending on the configuration it either uses QEMU (TCG/HVF) or Virtualization.framework. As you correctly assumed, when as a backend it uses QEMU, it can emulate various different architectures, for example it can emulate an x86_64 Linux VM from an ARM64e host, or the other way around.

In our case, we are interested into virtualizing an ARM64e macOS VM from an ARM64e host (M1), and if that's the environment, UTM lets us use the Virtualization.framework backend as an alternative of QEMU, which instead of emulation it does proper virtualization.

The Virtualization.framework is provided by Apple and they implemented all the machinery to being able to virtualize a macOS VM, including but not limited to VCPUs, memory balloons, Interrupt Controllers, etc, using hardware accelerations (dispatching instructions as-is to the CPU without emulation layers).

The resulting VM will run in EL1 while the host is running in EL2.

To answer you question about the "running copies of the host version": no, VMs are not copies of the host OS, they are they own OS running, each having its copy of the kernel, users-pace, etc. The reason why they say that only "macOS Monterey or higher" is supported is because the Virtualization.framework supports macOS guests only from Monterey.

I guess that this also answers the question about the kernel panics, which is: if a VM kernel panics, only that VM will be impacted, and no other components on the system (for example the host) will reboot. So yes it's suitable for kext development, and general kernel research where you want only the guest to be panicked/rebooted and not the host.

I hope this is answering your questions, but feel free to elaborate them if you still have some doubts.

[steven-michaud]

Thanks for the information. I discovered on my own that Apple's Virtualization framework docs include sample code for "Running macOS in a Virtual Machine on Apple Silicon Macs". This shows that the Virtualization framework runs its VMs from "VM Bundles" stored in your home directory. So I was wrong in my speculation that the Virtualization framework uses something like chroot to run its VMs. It uses images stored on disk, just like VMware Fusion.

There are more docs on "configuring and connecting devices to guest operating system instances", including block storage devices for the guest system.

When the time comes, I should be able to use these docs to figure out how to set up a UTM VM that uses the Virtualization framework. I may also try to set up one that uses QEMU -- though I suspect I'll quickly run into the Hackintosh problem doing that.

Once again, it'll be a while before I do this. I'd still like to wait for VMware and/or Parallels to provide support for macOS clients on Apple Silicon. But if/when I get tired of waiting, it looks like I (probably) have a viable alternative in UTM.

[steven-michaud]

I've started playing with UTM earlier than I expected (running a macOS 12 Apple Silicon client on a macOS 12 Apple Silicon host). I'm a bit surprised at how well it works, particularly the bridged networking. But there's still something missing, which will completely block my attempts to do any kernel extension development on it: There's no way to boot into a recovery partition in the guest, to turn off SIP.

UTM has an open issue on this. I tried to work around the problem by using createinstallmedia in the macOS 12.3.1 full installer to create a bootable installer on a second partition or hard drive. In VMware Fusion guest VMs I've been able to boot from this to disable SIP. But my efforts were blocked in two different ways:

1. At least in macOS 12.3.1 on Apple Silicon, createinstallmedia fails to bless a non-external hard drive or partition.

2. And if I use createinstallmedia to create a bootable external USB drive in the host, I'm unable to attach this drive to the guest.

[steven-michaud]

Another way to boot into a recovery partition in a UTM macOS VM would be to

1. Use createinstallmedia to create a bootable ISO.

2. Add the ISO as an external drive.

3. Move it up in the boot sequence, then boot the VM.

Steps 1 and 2 work fine, but step 3 doesn't. I've opened a UTM issue on this.

[steven-michaud]

I've opened a UTM issue on this.

I've found a workaround for this issue. It'll still be a while before I start working on porting HookCase to Apple Silicon. But at least now I have a way forward ... or so it seems.

[steven-michaud]

But at least now I have a way forward

No, I don't. I can now run the Recovery utility in UTM's Apple Silicon guest, and change its SIP and other security settings. But a plain-vanilla kernel extension won't load, even at the most permissive security settings. It loads just fine in the host OS (macOS 12.4). This issue has already been reported at UTM. I don't know of any workaround.

[dariaphoebe]

Reported to Apple as FB10145502 (third party kexts cannot be used in Virtualization framework apps)

[ccp10841054]

Hookcase upported architecture Apple Silicon M1 Now?
when turn on Apple's System Integrity Protection.

[steven-michaud]

No. The situation hasn't changed since my last comment above. Apple's Virtualization Framework still doesn't support loading non-Apple kernel extensions in "guests" (though they load just fine in the "host"). For more information see utmapp/UTM#4026.

[ccp10841054]

I have developed a dynamic library myself and want to inject it into other third-party applications (such as Safari, Google Chrome and pages). However, Mac OS 11. X adds many restrictions, such as SIP and sign. Resulting in injection failure.
Then I saw your hookcase. I want to ask, can hookcase meet my needs？

[steven-michaud]

HookCase doesn't (yet) work on Apple Silicon hardware. If you have an Intel Mac running macOS 12 or below, please try it out. And do read the documentation. But it assumes you already have some basic knowledge -- for example of what hooks are. If you don't know this, you probably can't do much with HookCase even on Intel hardware.

[ccp10841054]

@steven-michaud
Steven, thank you for your reply! Sorry, I have read the document, but my English is not good,so I can't understand some places. I'm very sorry for the trouble
I tried to send you an email asking for face-to-face communication through instant software, but failed

“If you have an Intel Mac running macOS 12 or below, please try it out” ------------ I'll try

In addition,Apple forbidden symbols:
kauth_ listen_ scope、sflt_ register、sflt_ unregister
I need to listen to these symbols and then complete my injection. I don't know whether additional processing is required

[steven-michaud]

I'm sorry, but I can't help you. If you can't read HookCase's documentation, you won't be able to use it.

I spent a lot of time making the docs as clear and straightforward as possible. If I were to try to help you here, or in email, I'd just be repeating their contents back to you. There's no point in doing that.

I realize that having the documentation only available in English is a problem. But HookCase is a small-scale project, and I don't have the resources to translate it into other languages.

[ccp10841054]

@steven-michaud Thank you for your patience and guidance
No need to translate hookcase documents into other languages, which is troublesome and time-consuming。

I can understand most of them. Because the project is relatively fast, I want to communicate with you and get the answers I want as soon as possible, proving that it is at least feasible. Otherwise, my later work will be in vain.

So I still want to confirm the following questions:

Apple Documents：Kernel Extensions have been deprecated. Starting with macOS Big Sur, macOS releases no longer load kernel extensions that use deprecated kernel programming interfaces (KPIs) by default.

• 1）About load kernel
  • Hookcase bypasses this restriction and implements kernel loading On Intel hardware for MacOS 12 or below, but just only Intel hard, kernel loading doesn't (yet) work on Apple Silicon hardware.------Is that right？？
  • Boot args must be Change for the first load kernel(sudo nvram boot-args="keepsyms=1" ). and to do this,SIP must be disabled.------Is that right？？
  • Kernel loading: replace kextload with kmutil load. ------Is that right？？
• 2）About deprecated kernel API
  such as :kauth_ listen_ scope、sflt_ register、sflt_ unregister.Hookcase bypasses this restriction,I can use this API On Intel hardware for MacOS 11 or above。------Is that right？？
• 3）About DYLD_INSERT_LIBRARIES
  About DYLD_INSERT_LIBRARIES are the same On all MacOS versions on Intel ，Is that right？？

I am a freshman about kernel.I hope to have a chance to meet you, a technical specialist。Thank you!!!!!!!!!

[panjiancai]

Very much looking forward to supporting Apple chip, please take some time to solve it, thank you

[myrontan]

@steven-michaud
Any progress? Thanks!!!

[steven-michaud]

@steven-michaud Any progress? Thanks!!!

No, basically.

Apple still doesn't support loading third-party kernel extensions in its macOS VMs on Apple Silicon -- not even on the macOS 14 betas. I've tried hard to find a workaround, without any success. See utmapp/UTM#4026 for more information.

Hang on to your Intel Macs! And not just for HookCase. Apple's support for macOS VMs on Apple Silicon is still very limited, and is unlikely ever to be as good as the support provided by third parties like VMware on Intel Macs.

[steven-michaud]

After months of hard work, I've discovered a workaround for Apple's design flaw: Running Third Party Kernel Extensions on Virtualization Framework macOS Guest VMs.

So now, finally, I can start work on porting HookCase to Apple Silicon. There's still no guarantee of success, and no promise that I won't run into further roadblocks from Apple. But the way seems clear, at least for the time being.

This will entail many more months of hard work -- at least six months. But there's now hope that I'll be able to resolve this bug sometime next year.

[ni-max]

@steven-michaud please check this approach https://github.com/LIJI32/MIP does it help?

[jayli123]

@steven-michaud May I ask, how can I port the hook sysent from my project to Apple Silicon? Your help would be greatly appreciated.

[steven-michaud]

@steven-michaud May I ask, how can I port the hook sysent from my project to Apple Silicon? Your help would be greatly appreciated.

You can't do this with HookCase on Apple Silicon, because HookCase doesn't (yet) support Apple Silicon.

After I found a way to load third-party kernel extensions in a Virtualization framework macOS guest VM, I quickly made progress towards porting HookCase to Apple Silicon. But then, towards the end of last year, I ran into a roadblock that I still haven't overcome. I've also had to deal with a fairly serious health problem -- a bad infection. And though the infection is now completely healed, I haven't yet resumed my porting work.

I will again, in the not too distant future. It may help that I've let the problem lie fallow for so long -- with luck I'll be able to approach it with an open mind. But I have no idea when, or whether, my efforts will succeed.

[steven-michaud]

@steven-michaud please check this approach https://github.com/LIJI32/MIP does it help?

Sorry for taking so long to respond to your suggestion, @ni-max. I just looked (briefly) at this project, and it does seem interesting. I particularly like that it's so well documented. But it seems to take a radically different approach from HookCase. It doesn't seem to use hooks at all. It just injects code. I'll look at it more closely when time permits. But that might not be for quite a while.

[jayli123]

@steven-michaud please check this approach https://github.com/LIJI32/MIP does it help?

Sorry for taking so long to respond to your suggestion, @ni-max. I just looked (briefly) at this project, and it does seem interesting. I particularly like that it's so well documented. But it seems to take a radically different approach from HookCase. It doesn't seem to use hooks at all. It just injects code. I'll look at it more closely when time permits. But that might not be for quite a while.

is there any progress

[steven-michaud]

is there any progress

As you can see from the comments above, the answer is "no". There's no point in asking again for at least another six months.