ARM builds?

[takluyver]

This is a very speculative issue. I'm aware that PEP 513 says manylinux1 is only defined for i686 and x86_64. But it would be nice if instructions telling people to pip install compiled modules worked easily and quickly on things like the Raspberry Pi too. It might also be valuable to the few brave souls trying to make Python work on mobile platforms.

So this issue is a place to work out what would be needed.

1. Architecture requirements: there are a lot of different versions of ARM, and there are confusingly similar looking version schemes for the architectures (e.g. ARMv7) and cores (e.g. ARM11). There may also be optional modules. Debian supports three buckets of ARM architectures: armhf requires ARMv7 and a floating point unit (hf = hardware float). armel supports older architectures, and cores without an FPU. arm64 is for the newer 64-bit processors. In terms of raspis:

• Raspi 0 and 1 have ARMv6 architecture, so Debian armhf does not support them. They do have FPUs, though, so Raspian is compiled to make use of this.
• Raspi 2 has ARMv7
• Raspi 3 has a 64-bit ARMv8 architecture.

2. Minimum library versions: These probably don't need to be all that old, because the systems I think this would target are likely to be running a relatively recent distro. But we'd need to work out what they are.

3. Distro for building: What distro would be the equivalent of the Centos 5.11 we use for building x86 and x64 manylinux wheels?

4. Build platform: Are there any CI services which offer ARM builds? Is virtualisation fast enough? Can we make it easy for people to build wheels on their own Raspberry Pi or similar device.

[rmcgibbo]

I am not personally very knowledgeable about ARM, so I don't know the direct answers to any of those questions. It would be really excellent if manylinux could support ARM at some point in the future though -- I have the impression that it's a growing platform.

For the manylinux1 spec, @njsmith and I leveraged a lot of Continuum's work to define the base distro and minimum library versions. Since the manylinux1 release, Continuum has added some support for ARM to conda and their packages, so perhaps this would be a good place to begin to identify best practices.

[takluyver]

Thanks, that's good to know. It looks like they're targeting ARMv7 and using Raspbian to run the builds.

[njsmith]

Unless someone wants to do lots of abi compatibility testing across different arm distributions, I think the most viable way forward here would be to push forward with the idea of distro-specific wheel tags, so that people could upload e.g "raspbian wheels"?

…

On Dec 12, 2016 3:40 AM, "Thomas Kluyver" ***@***.***> wrote: Thanks, that's good to know. It looks like they're targeting ARMv7 and using Raspbian to run the builds. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#84 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAlOaLss6R_-ibhkVU8Q45O-oTuK5bUhks5rHTKugaJpZM4LJ_pp> .

[KellenSunderland]

Using dockcross works quite well for ARM builds. I've created a few packages that use cross-compilation rather than virtualization. Builds that take hours on a Pi typically take a minute or two. This Dockerfile for example builds packages that run well for me with a Pi3 + Raspbian jessi. https://github.com/dockcross/dockcross/blob/master/linux-armv7/Dockerfile

[takluyver]

Since my motivation for this was getting Python packages on a Raspberry Pi, I'll drop a link to piwheels, @bennuttall's project to compile wheels for that platform.

[bennuttall]

An overview:

Although the hardware spec of Pi 1/Zero is ARMv6, Pi 2 is ARMv7 and Pi 3 is ARMv8, the reality is that Raspbian runs in ARMv6 user space regardless on your hardware. When you build a wheel on a Pi it's tagged armv6l on Pi 1/Zero and armv7l on Pi 2/3, but really it's always ARMv6.

So my project piwheels provides wheels for Raspberry Pi, but I wouldn't bank on the platform wheels working on other ARM boards. There's no particular need for maintainers to upload their wheels to PyPI as Pi users can use piwheels as their repo instead. Soon, in Raspbian, pip will be configured to use piwheels. (update: this is now true)

[QJKX]

Python on Termux (an android Linux environment) would also benefit from arm wheels.

[NotAFile]

We're having this problem with testing ARM in CI. The lack of wheels forces us to compile all dependencies for a fresh build, which takes around 20 minutes.

[vielmetti]

There is a CI system available, Drone Cloud, that can build natively for 32-bit and 64-bit Arm systems. There's also bare metal resources in the @WorksOnArm project to tackle this. Happy to help tackle this, especially for arm64 (aarch64).

[vielmetti]

@bennuttall - does piwheels support 32-bit Arm only, or do you also have 64-bit arm64 support?

[bennuttall]

We only provide wheels for 32-bit systems.

[vielmetti]

Referencing https://marcin.juszkiewicz.com.pl/2019/02/23/we-need-arm64-systems-for-developers-again/ which describes the resulting pain that pip install scipy, which takes a few seconds on x86, takes hours on arm64 because of lack of wheels.

[hrw]

I discussed that with Pypi devs some time ago on irc.

Whole service depends on whatever authors upload. So if we want non-x86 wheels then we need to convince authors to build and upload arm wheels.

[hrw]

And when it comes to what to use as a minimal...

I think that Debian jessie is sensible minimum. Eventually CentOS 7.

Older releases did not supported arm64.

[ralphie-boy]

I need to do python development that can include the python SpeechRecognition package. Samsung recently released the Galaxy Book 2 with the latest 64 bit Qualcomm Snapdragon processor (850). It comes with Windows 10 S which can be upgraded to Windows Pro. Windows Pro will support WSL (Windows Subsystem for Linux). WSL will allow me seamlessly run Ubuntu. Ubuntu will have built-in Python. The question is what happens when I try to install the SpeechRecognition package? I know for example this will not work on iPad. Will the Arm 64 architecture support this package?

[ax3l]

If the manylinux project releases official images for a platform, there is a hidden assumption that is because the community expects library authors to support it and use it to upload images to PyPI.

Hm, I understand the pain but I am not sure I can follow the "signaling/assumption" reasoning. Wouldn't manylinux-related PEPs, such as defining platform tags for big endian ppc64 or even allowed uploads to PyPI for exotic architectures, be signaling what you mention to developers?

I think we should define tags for even more architectures and micro-architectures and signal to the community via PyPI support/rejects in uploads. It's not bad to have more compatible manylinux base images, e.g. to drive a sub-community specific index or local builds.

[mattip]

It's not bad to have more compatible manylinux base images

Your position makes sense for pypa/manylinux as long as the people involved here have the bandwidth to continue to maintain the images. Help with gh-501, gh-542 and pypa/packaging#293 would be welcome.

I guess we (NumPy) will have to discuss internally how we respond to requests for additional wheels.

[ShadowJonathan]

This may be a digression, but are the changes that're suggested in pypi/warehouse#7853 accurate to pep 600 and the projected arm arches to be supported?

More specifically, this suggested addition.

[mayeut]

@ShadowJonathan,

warehouse maintainers requested that all architectures supported by PEP599 could be uploaded in pypi/warehouse#6684

pypi/warehouse#7853 only provides no-regression for those architectures and that doesn't mean there will ever be an armv7l image.

As @mattip stated, the lack of known open-source CI support is a major show stopper. Even with travis-ci supporting aarch64, ppc64le and s390x, we're seeing various issues on those platforms and, for the time being, I don't foresee expanding the supported architectures and, not before we have something acceptable working for PEP600.

@ax3l,

Support for even more architectures is allowed by PEP600. It's up to every project to allow anything.
As for manylinux base images, the previous comment on open-source CI support applies. There's also the the issue of toolchain support across architectures to be able to provide a single, consistent software environment for all architectures. This already became a hassle with i686 on CentOS with devtoolset dropping support of i686 and not being able to cross-compil for i686from x86_64 (that would also have been a hassle but perhaps less).

As for micro-architectures, this is really outside of the scope of manylinux and is a much more generic tagging issue applying to windows, linux, macOS, ...
My 2 cents anyway (not as a maintainer of manylinux) is that I'd rather see libraries relying on micro-architecture optimizations to provide proper dynamic dispatch mechanisms with fallbacks as is the case for example with OpenBLAS or libjpeg-turbo.

[sandys]

hey guys,
we would very much like to see ARM builds happen because of the general availability of AWS Graviton 2 instances on Amazon Cloud.
At this point, they are best ROI for a cloud dollar. https://aws.amazon.com/ec2/graviton/ and https://www.anandtech.com/show/15578/cloud-clash-amazon-graviton2-arm-against-intel-and-amd/10

Being able to have our stack natively on ARM would cut down costs immensely. And I think, this would apply to the majority of people out there who use AWS.
Also, given Apple is going fully ARM, we see this trend accelerating and not slowing.

[lpsinger]

I very much agree. However, note that manylinux would serve the AWS use case but not the Apple one (same CPU architecture, different OS and executable format).

[takluyver]

To be clear, you don't need manylinux support to build and use Python extensions for a particular ARM platform. https://www.piwheels.org/ already does this for Raspberry Pi, for example, and you could do something similar for ARM on AWS. So long as you know a particular target platform, you can compile on the same platform.

Manylinux is meant to make it possible to build a wheel on Fedora with one ARM chip and then install it on Ubuntu with a different one - which is obviously useful, but also much harder.

[geoffreyblake]

This issue is pretty old and hasn't seen updates for ~6 months, but is there still debate on whether manylinux2014_aarch64 is stable and abiding by PEP 599?

I've set up a tester to automatically try some of the top wheels to see how Arm64 support is coming along using Graviton systems and the wheels built with manylinux2014_aarch64 appear to work well on both armv8.2 and armv8.0 systems running a cross-section of distros (Ubuntu 20.04, AL2, Centos8): https://github.com/geoffreyblake/arm64-python-wheel-tester

The failures seen by the tester for the packages tested are due to lack of wheels.

[mayeut]

Original post from @webknjaz in #1029

armv7l (that a lot of raspberries use because the official 64-bit kernel is still in beta) and ppc64 are present in the PEP but the images for these aren't built.

I haven't found any discussion mentioning them so I just wanted to have someplace documenting why they are missing.

My wild guess is that it's because Travis CI doesn't provide them. Any other reasons? It looks like it should be possible to build them with QEMU under GHA, no?

[mayeut]

@webknjaz,

Just to go a bit further, PEP600 which supersedes PEP599 doesn't restrict architectures at all.

The main reasons not to add those are:

• the lack of a base image (for ppc64)
• the lack of native CI
• debugging issues locally

QEMU does not solve everything. It's painfully slow and GHA builds are timing out now and then on ppc64le since I could not get them running back on travis-ci yet.
I've seen in the past and I'm still seeing crashes, only when running QEMU which prohibits me from debugging at all locally. I unfortunately do not have time to go through debugging/reporting/contributing/waiting for fixes in QEMU.

[webknjaz]

GHA builds are timing out now and then on ppc64le

Do you know you could increase the timeout per-job and per-step? https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idstepstimeout-minutes

[mayeut]

@webknjaz,
there's a hard limit per job: 6 hours. that's the one we're hitting.
we'd need to rewrite again all scripts to bypass this limit and anyway, debugging in CI is not very sustainable, debugging in CI with QEMU even less no ?

[MrNaif2018]

@mayeut What about using qemu on aarch64 machines? CircleCI has added arm64 machines support and it's pretty stable, and qemu emulation runs almost at the same speed as native builds (because I guess arm32 is supported by kernel or the spec), and they might add arm32 machines soon.
As for testing locally: I agree, testing locally on amd64 with qemu is painful, but what about testing on aarch64 machines?
Oracle cloud (as part with partnership with arm I think) provides always free 24 gb ram and 4 cpu arm machine:
https://docs.oracle.com/en-us/iaas/Content/FreeTier/freetier_topic-Always_Free_Resources.htm
So it's not fully local debugging, but pretty convenient (vscode can run via ssh too)
Would that help moving forward the armv7 wheels creation?

[Red-M]

I've just added such support for my own packages on arm64/aarch64 via using a full KVM to run the aarch64 docker containers for manylinux and its super slow. If you really want these, buy a bunch of RPi4s and run CI VMs on those via libvirt or KVM (hint, pimox). Otherwise piwheels exists, so make sure you provide an sdist.

Edit: To make this painfully clear, virtulized (x86 emulating arm) is incredibility slow (in my scenario, up to ~3 times slower for releasing a single build on the same hardware)

What would be nice is providing buildchains in the manylinux containers for other cpu architectures to allow for easy building without the need for actual hardware and to provide already cross-compiled python versions that already exist for the x86 binaries (eg, drop/supporting a version of python brings it to all supported architectures). This will make the containers larger but at least building wheels (eg, for arm) won't suck as much to do so.

No, cloud services do not count even if they are free. People need to be able to do this on any machine they own for reproduce-able builds (so the barrier to entry to confirmation is only running a docker container)

[mayeut]

What would be nice is providing buildchains in the manylinux containers for other cpu architectures to allow for easy building without the need for actual hardware and to provide already cross-compiled python versions that already exist for the x86 binaries (eg, drop/supporting a version of python brings it to all supported architectures). This will make the containers larger but at least building wheels (eg, for arm) won't suck as much to do so.

These do exists in some form outside of manylinux #1125 (comment) however, while it might be suitable for maturin / pyO3 where there is probably a decent/recent rust cross-compiler, it's not the case for the gcc toolchain as noted in that comment.

Moving this to discussion.