Building for IOS

[JackyCA]

I am currently at step 3 under "Building for iOS". I used rules_xcodeproj instead of Tusi to generate the project.

Having opened the xcworkspace with Xcode, in the Project navigator panel there is a 'Products' folder, which contains GDMP.framework, libacceleration_cc_proto.io, and a collection of .a files. The problem is that I cannot copy the library from the framework to your Godot project's addons/GDMP/libs/ios because the folder and the files are not visible from outside Xcode, like when I right-click and select "Show in Folder" it takes me to a folder without the folder/files. How do I proceed?

While we are on that, I would also like to ask for clarifications on step 4. What does exporting Godot project to Xcode mean? Does it refer to this?

It would be very helpful if you could share some screenshots.

Earnest thanks in advance.

[JackyCA]

I also tried to use Tulsi, but Tulsi complained about GDMP.tulsiproj not having a workspace.

[j20001970]

Sorry for late response, since Tulsi is no longer maintained, I suggest trying to build iOS static library with Bazel command directly. Open the terminal in the mediapipe directory, then run:
bazel(or bazelisk) build --config=ios-arm64 //GDMP/ios:GDMP

[JackyCA]

Thank you for your reply. This time after un-zipping I can access the GDMP.framework folder from Finder.

If I may ask a follow-up, is it correct if I copy GDMP.framework into addons/GDMP/libs/ios, then rename the GDMP (Unix Executable File) file inside to GDMP.a, then add the macos path in GDMP.gdextension and I am done? Because there is an error loading extension: res://addons/GDMP/GDMP.gdextension and it Can't open dynamic library: /absolute/path/to/addons/GDMP/libs/ios/GDMP.framework/GDMP.a. Error: dlopen [...].

Thank you again for your patience.

[j20001970]

Copying GDMP the unix executable file to addons/GDMP/libs/ios then rename to GDMP.a should be enough.

Since it is iOS library, the library entry name should be ios instead of macos, based on this .gdextension file example:

ios.debug = "res://bin/libgdexample.ios.template_debug.xcframework"
ios.release = "res://bin/libgdexample.ios.template_release.xcframework"

[JackyCA]

Sorry I could not reply over the last week.

I have followed your instructions and am now waiting for an App Store Team ID in order to export my project to Xcode (step 4 at long last). Meanwhile, may I ask if you can help me to dismiss some doubts.

1. Is it true that I will not be able to test my project on my mac as you mentioned the library is an iOS one. Rather, I will be connecting an iPad to my mac and exporting (remote debug) each time I test.
2. My unix executable file is ~1.93GB in size. When I execute it alone, it displays, roughly, the followings,

/fake/path/GDMP.framework/GDMP ; exit;
/fake/path/GDMP.framework/GDMP: line 1: syntax error near unexpected token 'newline'
/fake/path/GDMP.framework/GDMP: line 1: '!<arch>'

Saving session...
...copying shared history...
...saving history...truncating history files...
...completed.

[Process completed]

The terminal is then non-responsive. Is the executable correctly built?
3. I have tried the demo which you have very kindly set up for us. In particular the Android version. I did not interact there with a monolithic file such as GDMP.a here, instead I (or rather, you) FileAccess.open individual task_file. I am slightly confused: do I have to copy some .task file over to the iOS library or do I have to import the libraries using a different syntax?

My most sincere gratitude!

[j20001970]

Is it true that I will not be able to test my project on my mac as you mentioned the library is an iOS one. Rather, I will be connecting an iPad to my mac and exporting (remote debug) each time I test.

Yes, since it is the library built for iOS, you will need a device running iOS, or setting up a emulator.

My unix executable file is ~1.93GB in size. When I execute it alone, it displays, roughly, the followings,

The library should be loaded by Godot, not executing directly from macOS. When you export Godot project to iOS, corresponding native libraries specified by .gdextension will be exported along with XCode project.

do I have to copy some .task file over to the iOS library or do I have to import the libraries using a different syntax?

Copy .task and/or relevant files to Godot project should be enough, since GDMP has custom resource provider that leverage Godot's filesystem registered to MediaPipe. When you run graphs that use mediapipe::GetResourceContents to load resources such as models or assets, Godot's filesystem will be searched first, then fallback to platform-specific method.

[JackyCA]

Thank you for the detailed explanation. I am now at the point where I can export the project from Godot to Xcode and from Xcode to iPad.

However, I encountered an error while building and running remotely from Xcode onto my iPad. I found the followings in Report navigator > myProject > Run > Console,

USER ERROR: Can't open dynamic library: addons/GDMP/libs/ios/GDMP.a. Error: dlopen(/private/var/containers/Bundle/Application/40...8/test.app/Frameworks/GDMP.framework, 0x0002): tried: 
'/usr/lib/system/introspection/GDMP.framework' (no such file, not in dyld cache), 
'/private/var/containers/Bundle/Application/40...8/test.app/Frameworks/GDMP.framework' (no such file), 
'/private/preboot/Cryptexes/OS/private/var/containers/Bundle/Application/40...8/test.app/Frameworks/GDMP.framework' (no such file), 
'/private/var/containers/Bundle/Application/40...8/test.app/Frameworks/GDMP.framework' (no such file).
  at: open_dynamic_library (platform/ios/os_ios.mm:260)

In short, it cannot find GDMP.framework anywhere (which is kind of strange given that I renamed it to GDMP.a); and

USER ERROR: Failed loading resource: res://addons/GDMP/GDMP.gdextension. Make sure resources have been imported by opening the project in the editor at least once.
   at: _load (core/io/resource_loader.cpp:274)

, which also appears strange to me because it is a .app not a Godot project.
What went wrong?

Conceivably, with those errors, all MediaPipeXXX classes and such raise errors at runtime. But I also realize that these MediaPipeXXX are not recognized in the Godot Editor. Is that expected?

Thank you very much!

[JackyCA]

That was probably a mistake on my part. The force_load command was not there when I double-checked.

Now that I have added the force-load command, I have the following error instead,

Link test (arm 64)
  Building for 'iOS', but linking in object file (test/addons/GDMP/libs/ios/GDMP.a[2](gdlibrary.o) built for 'iOS-simulator'
	Linker command failed with exit code 1 (use -v to see invocation)

I tried setting Build Active Architecture Only to Yes and setting Excluded Architectures to Any iOS Simulator SDK but the error persists. How do I fix it?

Sorry for the trouble.

[j20001970]

I have no required setup to help you test that at the moment, but did you build the library with --config=ios-arm64? IIRC bazel defaults to simulator when building iOS library.

[JackyCA]

Indeed. When I tried config=ios-arm64, bazel quit with ERROR: Config value 'ios-arm64' is not defined in any .rc file. Not knowing what bazelrc to use and presuming that it would be OK as long as I had platform_type="ios" in the target BUILD file, I left it out - simulator never crossed my mind.

I made shift with --config=ios and --cpu=ios_arm64. I think Xcode now builds a bit further, but I still have some issues:

1. The fatal error Undefined symbols, the details of which I have put in the bottom for readability.
2. The warning Object file was built for newer 'iOS' version (15.0) than being linked (11.0). I had similar problems when I was using rules_xcodeproj. It was then 'value' is unavailable: introduced in iOS 12.0, which was the reason I am building for 15.0. I followed online comments to add a -w flag to eliminate the warning. Should I?

Thank you again. Your insightfulness was incredible as ever! I wish you and your team the best of luck.

---

Undefined symbols upon expansion reads,
ld: Undefined symbols: _OBJC_CLASS_$_ALAssetsLibrary, referenced from: in GDMP.a[620](cap_ios_video_camera.o) _kvImage_ARGBToYpCbCrMatrix_ITU_R_709_2, referenced from: (anonymous namespace)::vImageConverterForCVPixelFormats(unsigned int, unsigned int, long*) in GDMP.a[846](utiq.o) (anonymous namespace)::vImageConverterForCVPixelFormats(unsigned int, unsigned int, long*) in GDMP.a[846](utiq.o) _vImageCVImageFormat_Create, referenced from: (anonymous namespace)::vImageConverterForCVPixelFormats(unsigned int, unsigned int, long*) in GDMP.a[846](utiq.o) (anonymous namespace)::vImageConverterForCVPixelFormats(unsigned int, unsigned int, long*) in GDMP.a[846](utiq.o) _vImageCVImageFormat_Release, referenced from: (anonymous namespace)::vImageConverterForCVPixelFormats(unsigned int, unsigned int, long*) in GDMP.a[846](utiq.o) (anonymous namespace)::vImageConverterForCVPixelFormats(unsigned int, unsigned int, long*) in GDMP.a[846](utiq.o) _vImageConvert_AnyToAny, referenced from: vImageGrayToBGRA(vImage_Buffer const*, vImage_Buffer*) in GDMP.a[846](utiq.o) _vImageConverter_CreateForCGToCVImageFormat, referenced from: (anonymous namespace)::vImageConverterForCVPixelFormats(unsigned int, unsigned int, long*) in GDMP.a[846](utiq.o) _vImageConverter_CreateForCVToCGImageFormat, referenced from: (anonymous namespace)::vImageConverterForCVPixelFormats(unsigned int, unsigned int, long*) in GDMP.a[846](utiq.o) _vImageConverter_CreateWithCGImageFormat, referenced from: (anonymous namespace)::vImageConverterForCVPixelFormats(unsigned int, unsigned int, long*) in GDMP.a[846](utiq.o) _vImagePermuteChannels_ARGB8888, referenced from: CallFrameDelegate(void*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, MPPPacketType, mediapipe::Packet const&) in GDMP.a[367](MPPGraph.o) CreateCVPixelBufferForImageFrame(std::__1::shared_ptr<mediapipe::ImageFrame>, bool) in GDMP.a[846](utiq.o) CreateImageFrameForCVPixelBuffer(__CVBuffer*, bool, bool) in GDMP.a[846](utiq.o) clang: error: linker command failed with exit code 1 (use -v to see invocation)

[j20001970]

It took some time to setup a working environment for troubleshoot iOS libraries building issues, thanks for your patience.

Could you try out commit e608a0e with updated instructions? You may also want to update addons/GDMP/GDMP.gdextension in your Godot project for newly added iOS library entries.

[JackyCA]

I was on holiday. Sorry for replying late.

I have cloned a fresh ios-fix branch and followed the new instructions to the point of building on Xcode, where I encountered some new issues. But before that, I want to make sure I "extract the content of mediapipe/bazel-bin/GDMP/ios/GDMP.xcframework.zip to Godot project's addons/GDMP/libs" correctly.

I extracted mediapipe/bazel-bin/GDMP/ios/GDMP.xcframework.zip and copied the entire GDMP.xcframework folder into addons/GDMP/libs. Then when I built from Xcode, the Build process was uncannily fast (despite being successful) and the app failed to install because /GDMP.framework/Info.plist was non-existent,

Failed to install the app on the device.
Domain: com.apple.dt.CoreDeviceError
Code: 3002
User Info: {
    IDERunOperationFailingWorker = IDEInstallCoreDeviceWorker;
    NSURL = "file:///fake/path/Library/Developer/Xcode/DerivedData/005_Godot_2-dzpxmncgukivuabxwmpggdwgngtr/Build/Products/Debug-iphoneos/005_Godot_2.app/";
}
--
Unable to Install “005_Godot_2”
Domain: IXUserPresentableErrorDomain
Code: 1
Failure Reason: Please try again later.
Recovery Suggestion: Failed to load Info.plist from bundle at path /var/installd/Library/Caches/com.apple.mobile.installd.staging/temp.92vyxG/extracted/005_Godot_2.app/Frameworks/GDMP.framework; Extra info about "/var/installd/Library/Caches/com.apple.mobile.installd.staging/temp.92vyxG/extracted/005_Godot_2.app/Frameworks/GDMP.framework/Info.plist": Couldn't stat /var/installd/Library/Caches/com.apple.mobile.installd.staging/temp.92vyxG/extracted/005_Godot_2.app/Frameworks/GDMP.framework/Info.plist: No such file or directory
--
Failed to load Info.plist from bundle at path /var/installd/Library/Caches/com.apple.mobile.installd.staging/temp.92vyxG/extracted/005_Godot_2.app/Frameworks/GDMP.framework; Extra info about "/var/installd/Library/Caches/com.apple.mobile.installd.staging/temp.92vyxG/extracted/005_Godot_2.app/Frameworks/GDMP.framework/Info.plist": Couldn't stat /var/installd/Library/Caches/com.apple.mobile.installd.staging/temp.92vyxG/extracted/005_Godot_2.app/Frameworks/GDMP.framework/Info.plist: No such file or directory
Domain: MIInstallerErrorDomain
Code: 35
User Info: {
    FunctionName = "-[MIBundle _validateWithError:]";
    LegacyErrorString = PackageInspectionFailed;
    SourceFileLine = 64;
}

Thank you again for your support.

[j20001970]

the app failed to install because /GDMP.framework/Info.plist was non-existent

Could you try changing GDMP.xcframework to Do Not Embed in exported Godot Xcode project?

[JackyCA]

That completed the installation, and the app is now runnable.

However, when I ran the app, Xcode console showed the following error, very similar to one before

USER ERROR: Can't open dynamic library: addons/GDMP/libs/GDMP.xcframework. Error: dlopen(/private/var/containers/Bundle/Application/E042B713-CEC8-4353-9AF2-9062CD031039/005_Godot_2.app/Frameworks/GDMP.framework, 0x0002): tried: '/usr/lib/system/introspection/GDMP.framework' (no such file, not in dyld cache), '/private/var/containers/Bundle/Application/E042B713-CEC8-4353-9AF2-9062CD031039/005_Godot_2.app/Frameworks/GDMP.framework' (no such file), '/private/preboot/Cryptexes/OS/private/var/containers/Bundle/Application/E042B713-CEC8-4353-9AF2-9062CD031039/005_Godot_2.app/Frameworks/GDMP.framework' (no such file), '/private/var/containers/Bundle/Application/E042B713-CEC8-4353-9AF2-9062CD031039/005_Godot_2.app/Frameworks/GDMP.framework' (no such file).
   at: open_dynamic_library (platform/ios/os_ios.mm:260)
USER ERROR: GDExtension dynamic library not found: addons/GDMP/libs/GDMP.xcframework
   at: open_library (core/extension/gdextension.cpp:455)
USER ERROR: Failed loading resource: res://addons/GDMP/GDMP.gdextension. Make sure resources have been imported by opening the project in the editor at least once.

(It only strikes me as strange now that it complains about failing to open the library before about failing to load the .gdextension)

This time though I cannot force-load the dynamic GDMP.xcframework. How do I proceed?

[j20001970]

I think that's due to .xcframework built by Bazel doesn't have .a suffix for static library, which is required by Godot to determine whether the GDExtension library is dynamic or not. Can you rename the GDMP to GDMP.a inside .xcframework folder and build the app again, or try out ios-fix branch?

[JackyCA]

I have just renamed the GDMP unix executable to GDMP.a and, as a reminder, I am indeed working on the ios-fix branch. Now I have the following error,

dyld[479]: Library not loaded: /fake/path/Xcode/DerivedData/005_Godot_2-dzpxmncgukivuabxwmpggdwgngtr/Build/Intermediates.noindex/ArchiveIntermediates/005_Godot_2/InstallationBuildProductsLocation/Applications/005_Godot_2.app/Frameworks/GDMP.framework/GDMP
  Referenced from: <E0A1D0B2-2413-35B4-B2CF-0971ED70CCB7> /private/var/containers/Bundle/Application/3BD9F8E0-7373-427E-8D6A-79E9D59F1FA2/005_Godot_2.app/005_Godot_2
  Reason: tried: '/fake/path/Xcode/DerivedData/005_Godot_2-dz...tr/Build/Intermediates.noindex/ArchiveIntermediates/005_Godot_2/InstallationBuildProductsLocation/Applications/005_Godot_2.app/Frameworks/GDMP.framework/GDMP' (no such file)

I do not know if this helps, but the above error is logged in the run-time Console after a successful build; somehow the app is trying to access the library from Xcode, which is on my MacOS, instead of on the iOS device. The app freezes on the Godot logo screen.

Thank you again, especially when Bazel, MediaPipe, Tulsi / rules_xcodeproj, and Godot are all under such active development that a fix this day may be deprecated the other. Your continuous dedication has my deepest respect.

[j20001970]

Thanks for your reports. The error shows that it is still trying to dynamic linking GDMP.framework/GDMP, maybe just renaming the library is not enough, please rebuild the library on ios-fix branch again. The resulting GDMP.xcframework.zip should contain GDMP.a.xcframework, and change GDMP.gdextension library entry if needed.

[JackyCA]

I have rebuilt with a86c5e3. The resulting GDMP.xcframework.zip does contain GDMP.a.xcframework. GDMP is also no longer in Frameworks in Project Navigator in Xcode.

The app still freezes on the logo screen, but this time the Console says,

dyld[452]: symbol not found in flat namespace '_mediapipe_library_init'

[j20001970]

Did you modify GDMP.gdextension entries as well? If you copied GDMP.a.xcframework to Godot project's addons/GDMP/libs, please try change the ios.debug and ios.release to res://addons/GDMP/libs/GDMP.a.xcframework, or just rename the folder back to GDMP.xcframework

[JackyCA]

Sorry for replaying late.

After modifying GDMP.gdextension, I have

Undefined symbols for architecture arm64:
  "_vImagePermuteChannels_ARGB8888", referenced from:
      CreateCVPixelBufferForImageFrame(std::__1::shared_ptr<mediapipe::ImageFrame>, bool) in GDMP.a(utin.o)
      CreateImageFrameForCVPixelBuffer(__CVBuffer*, bool, bool) in GDMP.a(utin.o)
ld: symbol(s) not found for architecture arm64

Sorry to impose, but I have no idea how to "add Accelerate.framework to linked libraries" (in fact I have never worked on Accelerate in any context), so I want to ask for further instructions or online materials for reference.

[j20001970]

Under Targets -> General, there should be a list under Frameworks, Libraries, and Embedded Content, then click the + symbol to add Accelerate.framework

[JackyCA]

Thanks for the quick response. I put a simple animation into the main scene and it appears the app is looping correctly.

Is it expected for Godot to issue errors when exporting to Xcode? The previous error persists in Godot but nothing related appears in the Xcode Console. GDMP seems to be loaded correctly.

I will proceed to test if GDMP functions correctly -- almost there!

[JackyCA]

To avoid the Godot export error, I now take the two steps (Godot -> .xcodeproj and Xcode -> .ipa) both on Xcode, as described in the docs. Nevertheless, the app could not run GDMP. For example, the line

var task: MediaPipePoseLandmarker

gave the Console errors

USER SCRIPT ERROR: Parse Error: Could not find type "MediaPipePoseLandmarker" in the current scope.
   at: GDScript::reload (res://main.gd:3)
ERROR: Failed to load script "res://main.gd" with error "Parse error".
   at: load (modules/gdscript/gdscript.cpp:2788)

I cannot identify the source of the problem. Given the much longer installation time (~1 min), I think GDMP is in the app, only not loaded properly. Do I need to set anything (Iike Godot Project Settings > Plugins > GDMP) else to enable GDMP?
(The Godot Editor highlights MediaPipeXXX in red, if that is of any relevance)

[j20001970]

The error says MediaPipePoseLandmarker is not registered to Godot, it looks like GDMP didn't load successfully.

Is it expected for Godot to issue errors when exporting to Xcode? The previous error persists in Godot but nothing related appears in the Xcode Console.

Regarding to the error you previously mentioned, do you mean undefined _vImage* symbols, or _mediapipe_library_init?

[JackyCA]

Sorry for the confusion.

The error in that re-reply refers to the _vImage* error in the previous reply, which showed up in Godot's Output whenever I exported the project there. It appeared to me, following your screenshot example, I could eliminate errors from the Xcode building .ipa process but could not eliminate the aforementioned error from the Godot export project process.
(I also tried modifying the build settings in the exported .xcodeproj by adding Accelerate.framework and then retracing to re-export from Godot, hoping that Godot will also use the same build settings; it did not work).

Later I discovered that I can initiate both processes from Xcode. It seemed reasonable to me that this way both processes will use the build settings with Accelerate.framework. And as I built anew, although I did not find any _vImage* error in Xcode's Report Navigator, when I tried to use GDMP, the

MediaPipePoseLandmarker is not registered to Godot

error appeared, and I did not know what to do.

Thank you very much for your patience.

[j20001970]

Sorry for replying that late, I'm currently busy on other stuffs, while trying to get graphics acceleration to work in order to replicate the problem in the simulator (it is not quite usable without proper GPU support).

It might take some time to update you the progress, thank you again for your patience.

[JackyCA]

Don't worry. I am more grateful for you kind sharing than I can be upset about any delay!

[j20001970]

ios-fix branch has been updated, please try updated instructions again, there should not the need of additional configuration for exported Godot Xcode project this time.

We ended up taking the dynamic library framework approach for iOS, as MediaPipe tasks registered through self-registration would be stripped in final iOS app if we build the library statically.

[JackyCA]

That is exciting. I began some other undertakings though when you were working on the new updates. I will test and reply once I have finished those.