[Bug] Recent changes to homebrew cmake and python defaults break dev environment

[saosebastiao]

Describe the bug

Homebrew now defaults to installing Python 3.12, which breaks the PX4 MacOS dev environment.

Homebrew now follows PEP-668 regarding virtual environments. This means that it is expected to use virtual environments like venv or pipx in order to manage dependencies. PX4 should adopt these standards regardless, as there are many reasons why using global or user level dependencies instead of project level dependencies is a bad idea.

And to clarify, this means that the steps to manually install dependencies in the official documentation are broken, as well as the Tools/setup/macos.sh script.

To Reproduce

using homebrew:

brew tap PX4/px4
brew install px4-dev
python3 -m pip install --user pyserial empty toml numpy pandas jinja2 pyyaml pyros-genmsg packaging kconfiglib future jsonschema
make px4_sitl

Expected behavior

Build should succeed

Screenshot / Media

On python3 -m pip install --user pyserial empty toml numpy pandas jinja2 pyyaml pyros-genmsg packaging kconfiglib future the following error message is give:

error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try brew install
    xyz, where xyz is the package you are trying to
    install.
    
    If you wish to install a Python library that isn't in Homebrew,
    use a virtual environment:
    
    python3 -m venv path/to/venv
    source path/to/venv/bin/activate
    python3 -m pip install xyz
    
    If you wish to install a Python application that isn't in Homebrew,
    it may be easiest to use 'pipx install xyz', which will manage a
    virtual environment for you. You can install pipx with
    
    brew install pipx
    
    You may restore the old behavior of pip by passing
    the '--break-system-packages' flag to pip, or by adding
    'break-system-packages = true' to your pip.conf file. The latter
    will permanently disable this error.
    
    If you disable this error, we STRONGLY recommend that you additionally
    pass the '--user' flag to pip, or set 'user = true' in your pip.conf
    file. Failure to do this can result in a broken Homebrew installation.
    
    Read more about this behavior here: <https://peps.python.org/pep-0668/>

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
WARNING: Skipping /opt/homebrew/lib/python3.12/site-packages/numpy-1.26.4.dist-info due to invalid metadata entry 'name'

Flight Log

N/A

Software Version

main

Flight controller

sitl

Vehicle type

None

How are the different components wired up (including port information)

No response

Additional context

No response

[dagar]

Thanks @saosebastiao, would you be able to open a pull request fixing this in Tools/setup/macos.sh? Once that settles I can look at updating it for Linux.

[saosebastiao]

@dagar I'd love to take a stab at it. I'll let you know up front that I don't know anything about the codebase, dev practices, MR guidelines, etc. Is there a guide for submitting a first MR to the project? Any example MRs from a project fork (assuming that's how it is done here)?

Also, I don't know a lot about CMake and my own tinkering that led to this bug report leads me to believe we won't be able to do anything with python virtual environments unless we change the CMake build to get rid of the deprecated FindPythonInterp module and start using the recommended FindPython3 module, which allows CMake to use virtual environments via the Python_FIND_VIRTUALENV hint.

I can of course try to do that, but I might be in over my head with CMake, especially on a project that I'm unfamiliar with.

# Find Python
find_package(PythonInterp 3)
# We have a custom error message to tell users how to install python3.
if(NOT PYTHONINTERP_FOUND)
	message(FATAL_ERROR "Python 3 not found. Please install Python 3:\n"
		"    Ubuntu: sudo apt install python3 python3-dev python3-pip\n"
		"    macOS: brew install python")
endif()

option(PYTHON_COVERAGE "Python code coverage" OFF)
if(PYTHON_COVERAGE)
	message(STATUS "python coverage enabled")
	set(PYTHON_EXECUTABLE coverage run -p)
endif()

include(px4_config)
include(kconfig)
message(STATUS "PX4 config: ${PX4_CONFIG}")
message(STATUS "PX4 platform: ${PX4_PLATFORM}")

[DronecodeBot]

This issue has been mentioned on Discussion Forum for PX4, Pixhawk, QGroundControl, MAVSDK, MAVLink. There might be relevant details there:

https://discuss.px4.io/t/px4-sync-q-a-may-15-2024/38773/1

[saosebastiao]

In order to transition to a virtualenv environment, we do need to transition to the newer cmake FindPython3 system. I have a partially working branch in progress here:
https://github.com/saosebastiao/PX4-Autopilot/tree/fix_macos_homebrew_python_env

This seems to work for make px4_sitl, but it fails for anything on a nuttx platform. I can't seem to find what is going wrong.

[2/1211] Generating nuttx/mm/libmm.a
FAILED: NuttX/nuttx/mm/libmm.a /Users/danieltoone/ws/PX4-Autopilot/build/px4_fmu-v6xrt_default/NuttX/nuttx/mm/libmm.a 
cd /Users/danieltoone/ws/PX4-Autopilot/platforms/nuttx/NuttX/nuttx && /opt/homebrew/Cellar/cmake/3.29.3/bin/cmake -E remove -f /Users/danieltoone/ws/PX4-Autopilot/platforms/nuttx/NuttX/nuttx/mm/libmm.a && find mm -type f -name *.o -delete && make -C mm --no-print-directory --silent libmm.a TOPDIR="/Users/danieltoone/ws/PX4-Autopilot/platforms/nuttx/NuttX/nuttx" KERNEL=n EXTRAFLAGS= && /opt/homebrew/Cellar/cmake/3.29.3/bin/cmake -E copy_if_different /Users/danieltoone/ws/PX4-Autopilot/platforms/nuttx/NuttX/nuttx/mm/libmm.a /Users/danieltoone/ws/PX4-Autopilot/build/px4_fmu-v6xrt_default/NuttX/nuttx/mm/libmm.a
clang: warning: optimization flag '-fno-strength-reduce' is not supported [-Wignored-optimization-argument]
warning: unknown warning option '-Wlogical-op'; did you mean '-Wlong-long'? [-Wunknown-warning-option]
warning: unknown warning option '-Wno-format-truncation' [-Wunknown-warning-option]
warning: unknown warning option '-Wno-maybe-uninitialized'; did you mean '-Wno-uninitialized'? [-Wunknown-warning-option]
warning: unknown warning option '-Wno-stringop-truncation'; did you mean '-Wno-string-concatenation'? [-Wunknown-warning-option]
warning: unknown warning option '-Wno-logical-op'; did you mean '-Wno-long-long'? [-Wunknown-warning-option]
warning: unknown warning option '-Wno-nonnull-compare' [-Wunknown-warning-option]
warning: unknown warning option '-Wno-old-style-declaration'; did you mean '-Wno-out-of-line-declaration'? [-Wunknown-warning-option]
In file included from mm_heap/mm_initialize.c:33:
/Users/danieltoone/ws/PX4-Autopilot/platforms/nuttx/NuttX/nuttx/mm/mm_heap/mm.h:207:1: error: static assertion failed due to requirement 'sizeof(struct mm_freenode_s) <= (1 << (4))': Error size for struct mm_freenode_s

static_assert(SIZEOF_MM_FREENODE <= MM_MIN_CHUNK,
^             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/Users/danieltoone/ws/PX4-Autopilot/platforms/nuttx/NuttX/nuttx/include/assert.h:81:27: note: expanded from macro 'static_assert'
#    define static_assert _Static_assert
                          ^
/Users/danieltoone/ws/PX4-Autopilot/platforms/nuttx/NuttX/nuttx/mm/mm_heap/mm.h:207:34: note: expression evaluates to '32 <= 16'
static_assert(SIZEOF_MM_FREENODE <= MM_MIN_CHUNK,
              ~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~
7 warnings and 1 error generated.
make[1]: *** [bin/mm_initialize.o] Error 1
[8/1211] Generating nuttx/sched/libsched.a
FAILED: NuttX/nuttx/sched/libsched.a /Users/danieltoone/ws/PX4-Autopilot/build/px4_fmu-v6xrt_default/NuttX/nuttx/sched/libsched.a 
cd /Users/danieltoone/ws/PX4-Autopilot/platforms/nuttx/NuttX/nuttx && /opt/homebrew/Cellar/cmake/3.29.3/bin/cmake -E remove -f /Users/danieltoone/ws/PX4-Autopilot/platforms/nuttx/NuttX/nuttx/sched/libsched.a && find sched -type f -name *.o -delete && make -C sched --no-print-directory --silent all TOPDIR="/Users/danieltoone/ws/PX4-Autopilot/platforms/nuttx/NuttX/nuttx" KERNEL=y EXTRAFLAGS=-D__KERNEL__ && /opt/homebrew/Cellar/cmake/3.29.3/bin/cmake -E copy_if_different /Users/danieltoone/ws/PX4-Autopilot/platforms/nuttx/NuttX/nuttx/sched/libsched.a /Users/danieltoone/ws/PX4-Autopilot/build/px4_fmu-v6xrt_default/NuttX/nuttx/sched/libsched.a

There is a lot of noise that happens after this error, then it reports the same thing for several other nuttx files.

make px4_fmu-v6xrt_default | grep FAILED

FAILED: NuttX/nuttx/mm/libmm.a /Users/danieltoone/ws/PX4-Autopilot/build/px4_fmu-v6xrt_default/NuttX/nuttx/mm/libmm.a 
FAILED: NuttX/nuttx/sched/libsched.a /Users/danieltoone/ws/PX4-Autopilot/build/px4_fmu-v6xrt_default/NuttX/nuttx/sched/libsched.a 
FAILED: NuttX/nuttx/libs/libc/libc.a /Users/danieltoone/ws/PX4-Autopilot/build/px4_fmu-v6xrt_default/NuttX/nuttx/libs/libc/libc.a 
FAILED: NuttX/nuttx/arch/arm/src/libarch.a /Users/danieltoone/ws/PX4-Autopilot/build/px4_fmu-v6xrt_default/NuttX/nuttx/arch/arm/src/libarch.a 
FAILED: NuttX/nuttx/net/libnet.a /Users/danieltoone/ws/PX4-Autopilot/build/px4_fmu-v6xrt_default/NuttX/nuttx/net/libnet.a 
FAILED: NuttX/nuttx/fs/libfs.a /Users/danieltoone/ws/PX4-Autopilot/build/px4_fmu-v6xrt_default/NuttX/nuttx/fs/libfs.a 
FAILED: NuttX/nuttx/drivers/libdrivers.a /Users/danieltoone/ws/PX4-Autopilot/build/px4_fmu-v6xrt_default/NuttX/nuttx/drivers/libdrivers.a 

[saosebastiao]

I noticed via this issue that nuttx is an older version and is using a custom px4 build system fork to use cmake which nuttx didn't have at the time. Could an upgrade here possibly help?