Add native arm64 build slice to all Windows dependencies #190

PatTheMav · 2023-06-05T13:45:05Z

Description

This is an exploratory PR for possible future native ARM64 builds on Windows.

Some dependencies cannot be built for ARM64 yet, others require specific workarounds, yet others can be built for ARM64 but themselves are too recent to be supported by OBS Studio.

This PR will stay in draft state until these and other issues are solved and are in a state that do not result in unnecessary support burden to current maintainers and supporters.

This PR is based on #186 and the PRs it depends on.

Motivation and Context

Due to Visual Studio 17 2022 having full support for ARM64 builds as well as clang- and Ninja-based builds, selecting the correct build architecture for MSVC builds does most of the work to achieve native builds, there are some caveats though:

libvpx requires disabling more advanced NEON features not made available by Microsoft's ARM64 implementation.
x264 requires an update of the gas-preprocessor it ships with - luckily FFmpeg provides a very recent version of it (the version shipped with x264 is 9 years old)
FFmpeg requires the same gas-processor to be made available in the PATH
svt-av1 has not been made available as it's not well optimised for ARM64 - dav1d would be a better alternative, but aom is "good enough" for now
Qt6 builds without any additional hacks or issues
LuaJIT has merged ARM64 and is available now
Python supports Windows ARM64 since version 3.11, alas OBS Studio does not support Python 3.11 at runtime yet (you could build OBS Studio with it, but people couldn't add its libraries at runtime)
mbedtls is still broken on ARM64 with version 3.5.1
libdatachannel requires at least mbedtls 3.4.0, so both are entirely broken on ARM64 right now
vpl is disabled for ARM64
Aja NTV2 is disabled for ARM64

How Has This Been Tested?

Tested on Windows 11 ARM64.

Types of changes

New feature (non-breaking change which adds functionality)

Checklist:

My code has been run through clang-format.
I have read the contributing document.
My code is not on the master branch.
The code has been tested.
All commit messages are properly formatted and commits squashed where appropriate.
I have included updates to all appropriate documentation.

driver1998 · 2024-01-11T01:08:04Z

What is the error exactly in building mbedtls? I saw upstream fixed similar issues multiple times.

PatTheMav · 2024-01-11T03:30:23Z

What is the error exactly in building mbedtls? I saw upstream fixed similar issues multiple times.

Compiling aes.c breaks with version 3.5.1:

error C1189: #error:  This header is specific to X86, X64, ARM64, and ARM64EC targets

This is possibly fixed by Mbed-TLS/mbedtls#8168, so we're waiting for MbedTLS to cut a new release with those fixes included.

tommyvct · 2024-01-17T05:12:05Z

This needs to be rebased and merge #227 in.

RytoEX · 2024-01-17T05:30:45Z

This needs to be rebased and merge #227 in.

@PatTheMav and I work closely together off-thread. Rest assured, that we discuss this regularly. However, do note that this is being worked on as time and other priorities allow. Thank you for your understanding.

tommyvct · 2024-01-17T06:33:14Z

utils.pwsh/Install-BuildDependencies.ps1

Small nitpicks here. If you look a few lines above:

$Prefixes = @{ 'arm64' = ${env:ProgramFiles(arm)} 'x64' = ${env:ProgramFiles} 'x86' = ${env:ProgramFiles(x86)} }

This is actually incorrect. C:\Program Files is reserved for the programs that is native to this machine, or at least the same bitness. For example, an x64 version of vscode machine-wide installer will install vscode in C:\Program Files. In another word, both x64 and arm64 programs sits in C:\Program Files.

${env:ProgramFiles(arm)} is a different story, usually it maps to C:\Program Files (Arm) which is usually a place for arm32 programs, which is extremely rare to see these days.

tommyvct · 2024-01-17T12:20:49Z

together off-thread. Rest assured, that we discuss this regularly.

May I join the conversation?

RytoEX · 2024-01-17T15:17:17Z

together off-thread. Rest assured, that we discuss this regularly.

May I join the conversation?

I appreciate the enthusiasm, but these are conversations that we have either on voice calls or in ongoing DMs about getting stuff to work. It is mostly stuff like "so about that PR" and "here's an idea" and "here's how we might get that to work when we have time". The point is that he doesn't need a reminder to rebase this. He's aware. This is on our roadmap to address.

tommyvct · 2024-01-17T17:39:53Z

How about we start a separate thread in the development channel? Or I could just ask and @ both of you in the channel?

RytoEX

Just some quick observations. I did get a local Qt build working, but it requires setting both QT_QMAKE_TARGET_MKSPEC and QT_HOST_PATH. I've commented how I might approach this for CI. We can attempt to optimize it later.

RytoEX · 2024-01-18T15:43:26Z

deps.qt/qt6.ps1

    $CMakeTarget = @{
        x64 = 'x64'
        x86 = 'Win32'
+        arm64 = 'arm64'
    }

    $Options = ($Options -join ' ') -replace '-G Visual Studio \d+ \d+','-G Ninja' -replace "-A $($CMakeTarget[$Target])",''


Qt requires a working Qt host with precompiled tools (moc, uic, etc.). Here's a quick and dirty hack that got me working locally. We would probably have to have the arm64 job wait for a successful x64 run, download the build, extract it somewhere, and point the arm64 job at that Qt directory.

diff --git a/deps.qt/qt6.ps1 b/deps.qt/qt6.ps1 index a7aea57..f9ee2a9 100644 --- a/deps.qt/qt6.ps1 +++ b/deps.qt/qt6.ps1 @@ -146,6 +146,13 @@ function Configure { arm64 = 'arm64' } + if ( $Target -eq "arm64" ) { + $Options += @( + '-DQT_QMAKE_TARGET_MKSPEC:STRING=win32-arm64-msvc' + '-DQT_HOST_PATH:PATH=C:\Qt\6.6.1\msvc2019_64' + ) + } +

deps.windows/60-python.ps1

RytoEX · 2024-01-18T15:46:52Z

deps.windows/40-detours.ps1

-    [string] $Hash = 'e4bfd6b03e50de46b47abfbd1e46b384f0c5f833'
+    [string] $Hash = '734ac64899c44933151c1335f6ef54a590219221'


Is there a specific change needed between these hashes? Last time I asked about updating this, I was asked/told not to update from the stable tag, old as it may be.

Only the most recent tags support ARM64 at all in my tests.

FWIW, we are using the most recent tag, but there have been new commits since then.

RytoEX · 2024-01-18T15:47:21Z

deps.windows/40-luajit.ps1

-    [string] $Hash = '8635cbabf3094c4d8bd00578c7d812bea87bb2d3'
+    [hashtable] $Hashes = @{
+        x64 = '8635cbabf3094c4d8bd00578c7d812bea87bb2d3'
+        x86 = '8635cbabf3094c4d8bd00578c7d812bea87bb2d3'
+        arm64 = '43d0a19158ceabaa51b0462c1ebc97612b420a2e'
+    }


Will be obsoleted once we update this for all systems.

Should be obsoleted by 9654933 and ec04a29 (#228), which updated to a newer LuaJIT commit.

RytoEX · 2024-01-18T15:48:09Z

utils.pwsh/Setup-Target.ps1

-    $NumProcessors = (Get-CimInstance Win32_ComputerSystem).NumberOfLogicalProcessors
-
-    if ( $NumProcessors -gt 1 ) {
-        $env:UseMultiToolTask = $true
-        $env:EnforceProcessCountAcrossBuilds = $true
-    }
-


Was this moved somewhere, or is it no longer needed?

RytoEX · 2024-01-18T15:48:41Z

deps.ffmpeg/99-ffmpeg.ps1

@@ -130,11 +134,13 @@ function Build {
    }

    $Backup = @{
-        MSYS2_PATH_TYPE = $env:MSYS2_PATH_TYPE
+        MSYS2_PATH_TYPE = $Env:MSYS2_PATH_TYPE


Stray Env.

Suggested change

MSYS2_PATH_TYPE = $Env:MSYS2_PATH_TYPE

MSYS2_PATH_TYPE = $env:MSYS2_PATH_TYPE

RytoEX · 2024-01-18T15:52:51Z

deps.ffmpeg/30-libvpx.ps1

-    [string] $Uri = 'https://github.com/webmproject/libvpx/archive/refs/tags/v1.13.1.zip',
-    [string] $Hash = "${PSScriptRoot}/checksums/v1.13.1.zip.sha256"
+    [string] $Uri = 'https://github.com/webmproject/libvpx.git',
+    [string] $Hash = '1672f4db71370e07e43e1e121cf2e92bf97ea4ff'


v1.14.0-rc1 is out, so hopefully there's a stable tag that can be used soon.

I am trying the PR, but i get many errors for example:
1 git clone : https://github.com/PatTheMav/obs-deps/tree/windows-arm64-native, and checkout windows-arm64-native branch,
2 try to run it locally: .\Build-Dependencies.ps1 -Target 'arm64' -Configuration 'Debug' -PackageName 'ffmpeg'

3 many libraries are built successfully, but some libraries fail, for example, mbedtls,

1>Checking Build System
Building Custom Rule C:/tools/alpaca/obs/PatTheMav-obs-deps/windows_build_temp/mbedtls/library/CMakeLists.txt
asn1parse.c
aesce.c
aesni.c
aria.c
C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.36.32532\include\immintrin.h(15,1): fatal error C1189: #error: This header is spe
cific to X86, X64, ARM64, and ARM64EC targets [C:\tools\alpaca\obs\PatTheMav-obs-deps\windows_build_temp\mbedtls\build_arm64\library\mbedcrypto.vcxproj]
base64.c

another questions, I have no ideay how to build these modules

• libvpx: Uses shell support and make from MSYS2, compiled with MSBuild
• x264: Uses shell support and make from MSYS2, compiled with cl
• lame: Built natively using old (but still working) nmake script

@PatTheMav , could you guide me ?

thanks

#190 (comment)

RytoEX · 2024-04-26T20:53:02Z

Was just skimming back over this.

x264 requires an update of the gas-preprocessor it ships with - luckily FFmpeg provides a very recent version of it (the version shipped with x264 is 9 years old)

FFmpeg requires the same gas-processor to be made available in the PATH

This seems like it should be a trivial Merge Request on the x264 repo. Ideally, I'd like the change to happen there at some point.

svt-av1 has not been made available as it's not well optimised for ARM64 - dav1d would be a better alternative, but aom is "good enough" for now

VideoLAN's dav1d is a decoder only. It is not an encoder. Xiph's rav1e is an encoder.

PatTheMav force-pushed the windows-arm64-native branch 3 times, most recently from 4fe3845 to 3f37412 Compare September 19, 2023 20:25

PatTheMav force-pushed the windows-arm64-native branch from 3f37412 to 0ff2fee Compare November 20, 2023 19:19

PatTheMav force-pushed the windows-arm64-native branch from 0ff2fee to e18f0e7 Compare December 2, 2023 15:22

PatTheMav mentioned this pull request Dec 28, 2023

deps.ffmpeg: Switch Windows builds to native build toolchain #186

Merged

6 tasks

PatTheMav added 2 commits January 11, 2024 03:57

deps.ffmpeg: Add arm64 slice to FFmpeg build scripts for Windows

2d614ab

utils.pwsh: Replace x86 with arm64 detection for VSDevShell

51d8d3f

PatTheMav force-pushed the windows-arm64-native branch from e18f0e7 to 1587850 Compare January 11, 2024 02:58

PatTheMav force-pushed the windows-arm64-native branch 4 times, most recently from 618846f to 8e29188 Compare January 11, 2024 19:16

PatTheMav added 5 commits January 11, 2024 21:29

utils.pwsh: Add arm64 to target detection

c38e552

deps.windows: Add arm64 slice to dependency build scripts for Windows

2437dbf

deps.qt: Add arm64 slice to Qt6 build script

54731c5

CI: Add arm64 slice to Windows dependencies matrix

6efb75b

CI: Add arm64 slice to Windows Qt6 matrix

d831848

PatTheMav force-pushed the windows-arm64-native branch from 8e29188 to d831848 Compare January 11, 2024 20:29

DEBUG

58c4660

PatTheMav force-pushed the windows-arm64-native branch from 6f09be1 to 58c4660 Compare January 13, 2024 00:28

tommyvct reviewed Jan 17, 2024

View reviewed changes

RytoEX requested changes Jan 18, 2024

View reviewed changes

tommyvct mentioned this pull request Mar 12, 2024

Add support for Windows ARM64 #140

Draft

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add native arm64 build slice to all Windows dependencies #190

Add native arm64 build slice to all Windows dependencies #190

PatTheMav commented Jun 5, 2023 •

edited

driver1998 commented Jan 11, 2024

PatTheMav commented Jan 11, 2024

tommyvct commented Jan 17, 2024

RytoEX commented Jan 17, 2024

tommyvct Jan 17, 2024

tommyvct commented Jan 17, 2024

RytoEX commented Jan 17, 2024

tommyvct commented Jan 17, 2024

RytoEX left a comment

RytoEX Jan 18, 2024

RytoEX Jan 18, 2024

PatTheMav Jan 18, 2024

RytoEX Mar 27, 2024

RytoEX Jan 18, 2024

RytoEX Mar 27, 2024

RytoEX Jan 18, 2024

RytoEX Jan 18, 2024

RytoEX Jan 18, 2024

shijunz Jan 27, 2024

PatTheMav Jan 27, 2024

RytoEX commented Apr 26, 2024

		[string] $Hash = 'e4bfd6b03e50de46b47abfbd1e46b384f0c5f833'
		[string] $Hash = '734ac64899c44933151c1335f6ef54a590219221'

	MSYS2_PATH_TYPE = $Env:MSYS2_PATH_TYPE
	MSYS2_PATH_TYPE = $env:MSYS2_PATH_TYPE

Add native arm64 build slice to all Windows dependencies #190

Are you sure you want to change the base?

Add native arm64 build slice to all Windows dependencies #190

Conversation

PatTheMav commented Jun 5, 2023 • edited

Description

Motivation and Context

How Has This Been Tested?

Types of changes

Checklist:

driver1998 commented Jan 11, 2024

PatTheMav commented Jan 11, 2024

tommyvct commented Jan 17, 2024

RytoEX commented Jan 17, 2024

Choose a reason for hiding this comment

tommyvct commented Jan 17, 2024

RytoEX commented Jan 17, 2024

tommyvct commented Jan 17, 2024

RytoEX left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RytoEX commented Apr 26, 2024

PatTheMav commented Jun 5, 2023 •

edited