ARM builds?

[takluyver]

This is a very speculative issue. I'm aware that PEP 513 says manylinux1 is only defined for i686 and x86_64. But it would be nice if instructions telling people to pip install compiled modules worked easily and quickly on things like the Raspberry Pi too. It might also be valuable to the few brave souls trying to make Python work on mobile platforms.

So this issue is a place to work out what would be needed.

1. Architecture requirements: there are a lot of different versions of ARM, and there are confusingly similar looking version schemes for the architectures (e.g. ARMv7) and cores (e.g. ARM11). There may also be optional modules. Debian supports three buckets of ARM architectures: armhf requires ARMv7 and a floating point unit (hf = hardware float). armel supports older architectures, and cores without an FPU. arm64 is for the newer 64-bit processors. In terms of raspis:

• Raspi 0 and 1 have ARMv6 architecture, so Debian armhf does not support them. They do have FPUs, though, so Raspian is compiled to make use of this.
• Raspi 2 has ARMv7
• Raspi 3 has a 64-bit ARMv8 architecture.

2. Minimum library versions: These probably don't need to be all that old, because the systems I think this would target are likely to be running a relatively recent distro. But we'd need to work out what they are.

3. Distro for building: What distro would be the equivalent of the Centos 5.11 we use for building x86 and x64 manylinux wheels?

4. Build platform: Are there any CI services which offer ARM builds? Is virtualisation fast enough? Can we make it easy for people to build wheels on their own Raspberry Pi or similar device.

[rmcgibbo]

I am not personally very knowledgeable about ARM, so I don't know the direct answers to any of those questions. It would be really excellent if manylinux could support ARM at some point in the future though -- I have the impression that it's a growing platform.

For the manylinux1 spec, @njsmith and I leveraged a lot of Continuum's work to define the base distro and minimum library versions. Since the manylinux1 release, Continuum has added some support for ARM to conda and their packages, so perhaps this would be a good place to begin to identify best practices.

[takluyver]

Thanks, that's good to know. It looks like they're targeting ARMv7 and using Raspbian to run the builds.

[njsmith]

Unless someone wants to do lots of abi compatibility testing across different arm distributions, I think the most viable way forward here would be to push forward with the idea of distro-specific wheel tags, so that people could upload e.g "raspbian wheels"?

…

On Dec 12, 2016 3:40 AM, "Thomas Kluyver" ***@***.***> wrote: Thanks, that's good to know. It looks like they're targeting ARMv7 and using Raspbian to run the builds. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#84 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAlOaLss6R_-ibhkVU8Q45O-oTuK5bUhks5rHTKugaJpZM4LJ_pp> .

[KellenSunderland]

Using dockcross works quite well for ARM builds. I've created a few packages that use cross-compilation rather than virtualization. Builds that take hours on a Pi typically take a minute or two. This Dockerfile for example builds packages that run well for me with a Pi3 + Raspbian jessi. https://github.com/dockcross/dockcross/blob/master/linux-armv7/Dockerfile

[takluyver]

Since my motivation for this was getting Python packages on a Raspberry Pi, I'll drop a link to piwheels, @bennuttall's project to compile wheels for that platform.

[bennuttall]

An overview:

Although the hardware spec of Pi 1/Zero is ARMv6, Pi 2 is ARMv7 and Pi 3 is ARMv8, the reality is that Raspbian runs in ARMv6 user space regardless on your hardware. When you build a wheel on a Pi it's tagged armv6l on Pi 1/Zero and armv7l on Pi 2/3, but really it's always ARMv6.

So my project piwheels provides wheels for Raspberry Pi, but I wouldn't bank on the platform wheels working on other ARM boards. There's no particular need for maintainers to upload their wheels to PyPI as Pi users can use piwheels as their repo instead. Soon, in Raspbian, pip will be configured to use piwheels. (update: this is now true)

[QJKX]

Python on Termux (an android Linux environment) would also benefit from arm wheels.

[NotAFile]

We're having this problem with testing ARM in CI. The lack of wheels forces us to compile all dependencies for a fresh build, which takes around 20 minutes.

[vielmetti]

There is a CI system available, Drone Cloud, that can build natively for 32-bit and 64-bit Arm systems. There's also bare metal resources in the @WorksOnArm project to tackle this. Happy to help tackle this, especially for arm64 (aarch64).

[vielmetti]

@bennuttall - does piwheels support 32-bit Arm only, or do you also have 64-bit arm64 support?

[bennuttall]

We only provide wheels for 32-bit systems.

[vielmetti]

Referencing https://marcin.juszkiewicz.com.pl/2019/02/23/we-need-arm64-systems-for-developers-again/ which describes the resulting pain that pip install scipy, which takes a few seconds on x86, takes hours on arm64 because of lack of wheels.

[hrw]

I discussed that with Pypi devs some time ago on irc.

Whole service depends on whatever authors upload. So if we want non-x86 wheels then we need to convince authors to build and upload arm wheels.

[hrw]

And when it comes to what to use as a minimal...

I think that Debian jessie is sensible minimum. Eventually CentOS 7.

Older releases did not supported arm64.

[ralphie-boy]

I need to do python development that can include the python SpeechRecognition package. Samsung recently released the Galaxy Book 2 with the latest 64 bit Qualcomm Snapdragon processor (850). It comes with Windows 10 S which can be upgraded to Windows Pro. Windows Pro will support WSL (Windows Subsystem for Linux). WSL will allow me seamlessly run Ubuntu. Ubuntu will have built-in Python. The question is what happens when I try to install the SpeechRecognition package? I know for example this will not work on iPad. Will the Arm 64 architecture support this package?

[takluyver]

SpeechRecognition itself appears to be a pure Python package, so there should be no problem installing it anywhere Python works. But it looks like it's a wrapper around various local and remote speech recognition engines, so the actual difficulty will probably depend which backend you need. But it's almost certainly possible - you just might need to install some things through apt or compile them from source.

[ralphie-boy]

Thanks for the reply, takluyver. I think you are correct. But I failed to procure the Galaxy Book 2 in time for my trip. Instead I'm taking my iPad, on which, Pythonista is the only supported Python. So SpeechRecognition won't work, because Pythonista is not a true complete working Python. It's pretty good, but it is limited to a subset of packages for which dependencies can be supported by whatever wrappers.

[ulope]

On a slightly related note docker has supported multi-platform images for a while now through manifest lists and all official (i.e. made by docker) images on docker hub are multi platform.

Together with the new buildx multi platform support this could make providing ARM images much easer.

[takluyver]

Just to remind everyone who might contribute here: the essential first step would be to define appropriate ARM architecture tags. What instruction sets & extensions are relevant, what are the rules for compatibility (e.g. presumably a binary built with NEON extensions won't work on a CPU that doesn't have them), and how can we detect what the hardware supports?

If we can work that out, I'd be optimistic that we could do something for glibc-based distros on ARM relatively easily, based on what's already been done for x86 & x86_64.

An alternative approach would be to focus on defining more specific tags, e.g. for raspbian. But that's obviously not as broadly useful.

[hrw]

@takluyver for ARMv7 it should follow 'armhf' defined by Debian/Fedora/etc distros. No NEON, -vfp3d16.

AArch64 (aka arm64) is simpler - just gcc defaults.

And target Debian/jessie or Debian/stretch as minimal OS version and binaries should (imho) work under CentOS 7/8, Debian 8+, Ubuntu 16.04+ etc.

[lpsinger]

It would be more convenient if the arm manylinux image was based on the same OS as the x86_64 image. Some times I need to install additional build tools and it would create some friction if I had to use yum for x86_64 but apt for arm. Is there a CentOS port for ARM?

[geoffreyblake]

This issue is pretty old and hasn't seen updates for ~6 months, but is there still debate on whether manylinux2014_aarch64 is stable and abiding by PEP 599?

I've set up a tester to automatically try some of the top wheels to see how Arm64 support is coming along using Graviton systems and the wheels built with manylinux2014_aarch64 appear to work well on both armv8.2 and armv8.0 systems running a cross-section of distros (Ubuntu 20.04, AL2, Centos8): https://github.com/geoffreyblake/arm64-python-wheel-tester

The failures seen by the tester for the packages tested are due to lack of wheels.

[mayeut]

Original post from @webknjaz in #1029

armv7l (that a lot of raspberries use because the official 64-bit kernel is still in beta) and ppc64 are present in the PEP but the images for these aren't built.

I haven't found any discussion mentioning them so I just wanted to have someplace documenting why they are missing.

My wild guess is that it's because Travis CI doesn't provide them. Any other reasons? It looks like it should be possible to build them with QEMU under GHA, no?

[mayeut]

@webknjaz,

Just to go a bit further, PEP600 which supersedes PEP599 doesn't restrict architectures at all.

The main reasons not to add those are:

• the lack of a base image (for ppc64)
• the lack of native CI
• debugging issues locally

QEMU does not solve everything. It's painfully slow and GHA builds are timing out now and then on ppc64le since I could not get them running back on travis-ci yet.
I've seen in the past and I'm still seeing crashes, only when running QEMU which prohibits me from debugging at all locally. I unfortunately do not have time to go through debugging/reporting/contributing/waiting for fixes in QEMU.

[webknjaz]

GHA builds are timing out now and then on ppc64le

Do you know you could increase the timeout per-job and per-step? https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idstepstimeout-minutes

[mayeut]

@webknjaz,
there's a hard limit per job: 6 hours. that's the one we're hitting.
we'd need to rewrite again all scripts to bypass this limit and anyway, debugging in CI is not very sustainable, debugging in CI with QEMU even less no ?

[MrNaif2018]

@mayeut What about using qemu on aarch64 machines? CircleCI has added arm64 machines support and it's pretty stable, and qemu emulation runs almost at the same speed as native builds (because I guess arm32 is supported by kernel or the spec), and they might add arm32 machines soon.
As for testing locally: I agree, testing locally on amd64 with qemu is painful, but what about testing on aarch64 machines?
Oracle cloud (as part with partnership with arm I think) provides always free 24 gb ram and 4 cpu arm machine:
https://docs.oracle.com/en-us/iaas/Content/FreeTier/freetier_topic-Always_Free_Resources.htm
So it's not fully local debugging, but pretty convenient (vscode can run via ssh too)
Would that help moving forward the armv7 wheels creation?

[Red-M]

I've just added such support for my own packages on arm64/aarch64 via using a full KVM to run the aarch64 docker containers for manylinux and its super slow. If you really want these, buy a bunch of RPi4s and run CI VMs on those via libvirt or KVM (hint, pimox). Otherwise piwheels exists, so make sure you provide an sdist.

Edit: To make this painfully clear, virtulized (x86 emulating arm) is incredibility slow (in my scenario, up to ~3 times slower for releasing a single build on the same hardware)

What would be nice is providing buildchains in the manylinux containers for other cpu architectures to allow for easy building without the need for actual hardware and to provide already cross-compiled python versions that already exist for the x86 binaries (eg, drop/supporting a version of python brings it to all supported architectures). This will make the containers larger but at least building wheels (eg, for arm) won't suck as much to do so.

No, cloud services do not count even if they are free. People need to be able to do this on any machine they own for reproduce-able builds (so the barrier to entry to confirmation is only running a docker container)

[mayeut]

What would be nice is providing buildchains in the manylinux containers for other cpu architectures to allow for easy building without the need for actual hardware and to provide already cross-compiled python versions that already exist for the x86 binaries (eg, drop/supporting a version of python brings it to all supported architectures). This will make the containers larger but at least building wheels (eg, for arm) won't suck as much to do so.

These do exists in some form outside of manylinux #1125 (comment) however, while it might be suitable for maturin / pyO3 where there is probably a decent/recent rust cross-compiler, it's not the case for the gcc toolchain as noted in that comment.

Moving this to discussion.