Metal Support for iOS

[tomspilman]

This is an offspring of the discussion in #4571.

We need to implement a new Metal backend for iOS devices under our existing XNA style API. The tasks would include:

• Fix 2MGFX to support Metal shaders.
• Document how to write a Metal FX file.
• Port the stock effect FX files to Metal.
• Add our own bindings for just the Metal APIs we use (like we are for OGL, SDL, OAL).
• Implement a new graphics backend for Metal.
• Remove old OpenGL iOS code.
• Test on a bunch of new and old iOS devices.

[tomspilman]

So one big issue... Metal is not supported on all iOS devices.

Basically it is any iOS device with an A7 GPU or higher. This means iPhone 5c, iPhone 5, iPhone 4S, and iPhone 4 which make up over 30% of iPhones out there do not support Metal. Same sort of picture on the iPad side with lots of popular devices not supporting Metal.

This means we have two choices:

• Wait to implement Metal under XNA... it wasn't going to give us a big performance boost anyway.
• Figure out some way to support both Metal and OpenGL and not create a ton of additional work.

I'm leaning towards waiting myself.

[lithiumtoast]

See MetalBasic3D and MetalKitEssentials for two C# iOS examples using Xamarin.

@tomspilman As of right now, are there people using MonoGame with MonoTouch or does everyone going for iOS have to use Xamarin? Xamarin already has the C# API for Metal for both iOS and Mac available for use today. Do we still want to create our own bindings of Metal for Xamarin? (Need a license from Xamarin to get on the App Store)

[lithiumtoast]

Wait to implement Metal under XNA... it wasn't going to give us a big performance boost anyway.

@tomspilman Should probably wait until late 2016 or later then to integrate Metal with MonoGame iOS / Mac. Hopefully Apple says something about Vulkan by then and we figure out what the new MonoGame API will be to use the next generation graphics APIs.

[ronnycsharp]

Hi. Any news out there?
I am a bit nervous that all my applications based on monogame for the 3D-rendering part are not working anymore in the near future. Since I've heard that apple has deprecated OpenGL ES - I didn't sleep well anymore.

Best regards
Ronny

[Jjagg]

Nothing new here. I bet Apple will hold off on dropping GL for a bit, there'd be outrage if they didn't.
But it is worrying that they deprecated it :/

[tomspilman]

I am worried about this as well. I expect first thing that will happen is Apple disallowing new games using OpenGL. That could happen in the next 6 to 12 months IMO. Then maybe a year later they could consider hiding OGL games on new Apple devices. Then maybe after that totally delisting OGL games.

But i could be 100% wrong on that.

We will be tackling this soon as we have a OGL game that needs the same fix.

[harry-cpp]

A better option should be to not use Metal, but instead use Vulkan and MoltenVK. Metal is only used on Mac and iOS and we have pretty much no one who likes using them from the core dev team.

[tomspilman]

I thought Apple was disallowing MoltenVK also thought it was a closed source paid project.

Seems like maybe all that has changed?

I'm good with a Vulkan implementation as i'm doing one now as well. So we're covered.

[mrhelmut]

MoltenVK has been updated to be compliant with the iOS cert (it was using non public surfaces but that's now fixed and games using MoltenVK are being accepted).

[ronnycsharp]

Thanks for your feedback. Over the last couple of days, I' ve tried a few things and I have found a good emergency plan for me. I tested a few frameworks and it seems that apple's SceneKit could fill the gap on iOS. In MG my 3D models are exported to X-Files and then compiled to XNB-Files with the pipeline tool. With SceneKit, I have to export the original 3D-Model to a COLLADA File Format (dae). These collada-files could easily rendered natively with SceneKit. It seams that SceneKit has a similar ligning system, like MG. To be platform independent we need a monogame-adapter for iOS, which is based on SceneKit. But perhaps the MoltenVK/Vulkan plan is still a better idea.

[Mindfulplays]

Is there an update on Metal? Looks like most other frameworks support Metal fully. The performance implications means Metal is quite important in iOS.

Is there any help needed here?

I am impressed WWDC21 (and 20) didn't mention anything about the depreciation surprisingly. But I doubt that will remain true in the coming months or year.

[mrhelmut]

FYI #7523 (comment)

[KUNGERMOoN]

Hi, this thread seems to be focused mainly on implementing Metal for iOS, which got me wondering if support for Metal on macOS is being planned. If that wasn't the case, my guess is it would mean that Mac would still run on OpenGL. That however, would mean anything requiring a version of OpenGL newer than 4.1 (for example Compute Shaders), wouldn't be possible, which is kinda worrying

[Mindfulplays]

@SimonDarksideJ @mrhelmut @harry-cpp I would like to take steps towards this.

Here is a high-level plan:

• Bring back iOS tests (interactive) - I've sent a PR Bring back tests for iOS, update to dotnet 6+, ensure works on device/simulator #8099
• Add basic rendering tests using OpenGL on iOS (in progress)
• Add incremental Metal support (basic Batch/Draw commands - ensure rectangle+texture work)
• I propose adding a MonoGame.Framework.iOS.Metal project (new) that can be swapped in (first by tests and by users who are willing to test it) while keeping the current .iOS be OpenGL-friendly.
• Ensure tests work with Metal
• Once this is stable, and if community approves we can make iOS.Metal->iOS and have folks test it out.
• Will need a path toward mgcb + metal shaders later (TODO, need some pointers here - Perhaps Molten/etc route to compile shaders from GLSL -> Metal using mgcb?)

[harry-cpp]

@Mindfulplays Thank you for taking an interest in this task.

We've actually been discussing Metal support internally over the last week and have decided that we are not gonna support Metal directly, and instead, we will be supporting it using Vulkan. Vulkan on Apple platforms is actually MoltenVK, which is a Vulkan to Metal conversion layer.

We also have a partial Vulkan implementation already and are working on merging it into the main repo, so there are some starting points for that as well :)

• Bring back iOS tests (interactive) - I've sent a PR Bring back tests for iOS, update to dotnet 6+, ensure works on device/simulator #8099
• Add basic rendering tests using OpenGL on iOS (in progress)

Awesome work, and much appreciated!

The graphics tests are currently not runnable on the GitHub Action bots, however, we are in the process of bringing back our old graphics test bots online. They should come back some time during December, and once back, I'l help with getting your PR merged and running on them.

[Mindfulplays]

@harry-cpp That's great news! MoltenVK should solve both Android/iOS. I wonder if adding tests on mobile might help validate that effort (I am in the process of doing so). @tomspilman if there is any help needed on the VK front I am happy to help.

I will take a look at other iOS/mobile stuff meanwhile (if there are specific ones that I can participate please let me know, not just mobile). Thanks again!

[squarebananas]

We've actually been discussing Metal support internally over the last week and have decided that we are not gonna support Metal directly, and instead, we will be supporting it using Vulkan. Vulkan on Apple platforms is actually MoltenVK, which is a Vulkan to Metal conversion layer.
We also have a partial Vulkan implementation already and are working on merging it into the main repo, so there are some starting points for that as well :)

Is this Vulkan implementation built around the Compute Shader PR (#7533) or is it a completely separate effort. I ask because that PR is based on the ShaderConductor PR (#7345) and a goal there was to set the groundwork for supporting Vulkan & Metal.

Right now ShaderConductor is only used for converting HLSL to GLSL/ESSL, but it can also output DXIL for DX12, SPIR-V for Vulkan, or MSL for Metal, so this can be considered a first step towards supporting those platforms.

Just asking in case work is being duplicated.

[Mindfulplays]

Hi Tom @tomspilman ,
I am wondering about a few things discussed in the board meeting (perhaps fodder for the next meeting?):

• Vulkan -> MoltenVk -> Metal seems like a roundabout way to reach parity on iOS (given that OpenGL on iOS already works well). Metal is fundamentally a different beast IMO, with a level of abstraction higher than that of Vulkan.

• Shader compilation from HLSL -> SPIR-V -> MSL seems like another roundabout way (there are tools to convert HLSL -> MSL)

• Is the cost of writing Metal-based GraphicsDevice + Shader support vs cost of using MoltenVk on iOS/macOS assessed? It seems to me the former is more straightforward and more beneficial (shaders, multi-threaded-ness, support for visionOS/etc and other Apple goodies including debuggability/performance) longer term.

• FNA (FNA3d) seems to have gone the route of MoltenVk so wondering the value of doing so in MonoGame...

I am also curious if the plan is to move all DesktopGL targets to be MoltenVk in the future? (Which seems totaly fine - but would 'DX' go through MoltenVK too?)

Mostly questions for you folks for the next board meeting.

thanks!

[mrhelmut]

Apple deprecated OpenGL a long time ago, and there is a valid chance that in the not-so-distant future they ban OpenGL from their certification process. We should move to Metal/Vulkan as soon as possible.

SPIR-V has the advantage to be an intermediate language that nearly all modern shader compilers can work with, and that includes all gaming consoles. There would be a great advantage to modernize the shading pipeline to have a SPIR-V intermediate layer. This would unify all platforms under the same flag and ease maintenance. Also, an HLSL to MSL compiler is likely to use SPIR-V internally anyway.

Using MoltenVK over Metal directly is a shortcut to reuse the upcoming desktop Vulkan target. It avoids the need to write an entirely new backend (with possibly its set of own bugs) for just one platform and ease maintenance by having only one backend to maintain for multiple targets. Though if anyone wishes to contribute a Metal implementation, it would be welcomed. It's the same principle that today with DesktopGL and Android sharing the same code, except that in the future DesktopVK will share its code with both Android and iOS to ease maintenance.

MoltenVK is a Metal implementation of Vulkan, it translates Vulkan calls into Metal ones. You can see it as a driver, it's not an API. We would only have to maintain a Vulkan backend and the use of MoltenVK will be transparent. MoltenVK and Metal are specific to the Apple ecosystem, it can't replace Direct3D.

[Mindfulplays]

Apple deprecated OpenGL a long time ago, and there is a valid chance that in the not-so-distant future they ban OpenGL from their certification process. We should move to Metal/Vulkan as soon as possible.

Can't +1 this enough.

SPIR-V has the advantage to be an intermediate language that nearly all modern shader compilers can work with, and that includes all gaming consoles. There would be a great advantage to modernize the shading pipeline to have a SPIR-V intermediate layer. This would unify all platforms under the same flag and ease maintenance. Also, an HLSL to MSL compiler is likely to use SPIR-V internally anyway.

Agreed, seems very reasonable.

Minor quibble: HLSL->MSL is possible with the new Apple Metal Shader Converter. DXC tools from MSFT (very useful for converting dxbc->dxil etc) which forms the basis for the metal shader converter.

Even MojoShader has support for MSL directly (no SPIR-V routing) - if we update libmojo* from the latest sources: Given that MG-FX format is essentially frozen in time - most of the yesteryear tools that produce valid DXIL can work with the new graphics APIs without much fuss.

FYI, I have an initial MG+Metal version I have been working on: metal shader converter/DXC tools integrate 'seamlessly' into the MGFXC pipeline (and the final metal programs for the default MG effects such as sprite/basic etc work with the MG codebase) and many of the Platform* calls are routed for the basic use-cases (basic blit/batching works well).

Using MoltenVK over Metal directly is a shortcut to reuse the upcoming desktop Vulkan target. It avoids the need to write an entirely new backend (with possibly its set of own bugs) for just one platform and ease maintenance by having only one backend to maintain for multiple targets. Though if anyone wishes to contribute a Metal implementation, it would be welcomed. It's the same principle that today with DesktopGL and Android sharing the same code, except that in the future DesktopVK will share its code with both Android and iOS to ease maintenance.

MoltenVK is a Metal implementation of Vulkan, it translates Vulkan calls into Metal ones. You can see it as a driver, it's not an API. We would only have to maintain a Vulkan backend and the use of MoltenVK will be transparent. MoltenVK and Metal are specific to the Apple ecosystem, it can't replace Direct3D.

If I understand correctly: First implement Vulkan implement as DesktopGL replacement (macOS/Win/Linux + Android). Once that works, MoltenVK for Metal could come in later. Is this a correct understanding?

Meanwhile, I have a Metal implementation for MG cooking up like I mentioned above - no promises yet - but it feels like I have a good understanding/resolution for the large unknowns on iOS/macOS/etc (shader compilation, for instance). I hope MoltenVK or Metal-raw replaces the current mobile-GL implementation soon.

[harry-cpp]

I see some misunderstandings is going on in regards to Vulkan on macOS.

You just write your app using Vulkan and compile your hlsl shaders to spir-v using dxc, thats it, no extra steps is needed for getting your app running on macOS. There is no macOS or iOS or Apple specific stuff that we will need to maintain for this graphics backend.

See https://vulkan-tutorial.com/Development_environment#page_MacOS on a bit more details.

[galad87]

MoltenVK has got some limitations, see the portability extension VK_KHR_portability_subset and the others difference in MoltenVK documentation. And besides the possible Metal bugs, there will be spirv-cross and MoltenVK bugs too.
Anyway, MoltenVK seems quite stable lately.

[Mindfulplays]

I have made progress in my Metal implementation:

• Created tests to compare OpenGL vs Metal (Add end-to-end test framework inspired from iOS interactive tests #8110 + adding some more)
• Changed mgfxc to use dxc + spirv-cross (thanks for the tip!): All default MG shaders now work (i.e. converted to Metal shaders). (DXC + SpirV-cross are both OSS / Apache2; also tried using icculus/mojoshader but that seems to be outdated).
• The "80%" Pareto use-case (at least to me?) of shader, texture, DrawUserIndexedPrimitives, blend state, scissor/etc are all wired up and rendering matches 1:1 with OpenGL for all the InteractiveTests. The shaders+Metal implementation work on iOS (phone/tablet + simulator) and (haven't verified) should work with mac catalyst /vision os too..
• Of course need to look at performance (beyond the obvious), clean up, missing functionality etc. (ETA ~week? 2 weeks? need to run the plan with some maintainers first to see if this is even a candidate for MG)
• At the very least, the mgfxc + tests are separable from the Metal stuff and would aid Vulkan-ization/DX12-ification work?

Putting this out there in case there is interest for this ?

@harry-cpp Yes, point taken. I am mostly talking about mac catalyst / Universal binaries which are moving in the direction of iOS - Metal-only with more stringent limitations being enforced by Apple over time (BTW MoltenVK is fine for mac catalyst).

[mrhelmut]

Closing this one in favor of centralizing discussions in #4571