Problem Building pyOptSparse from Source on macOS (Apple Silicon)

[mcmehrtens]

Please forgive me, I'm new to pyOptSparse and optimization in general. I'm trying to configure a virtual python env (venv) capable of running a script that uses OpenMDAO and SNOPT. I am referring to these instructions.

I am working on a 2021 M1 Pro MacBook Pro (Apple Silicon; ARM64). I have confirmed I have gcc, gfortran, and numpy installed (see below).

Installation Verification

(openmdao_env) Matthews-MacBook-Pro:pyoptsparse mcmehrtens$ gcc --version
Apple clang version 14.0.3 (clang-1403.0.22.14.1)
Target: arm64-apple-darwin22.4.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
(openmdao_env) Matthews-MacBook-Pro:pyoptsparse mcmehrtens$ gfortran --version
GNU Fortran (Homebrew GCC 12.2.0) 12.2.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

(openmdao_env) Matthews-MacBook-Pro:pyoptsparse mcmehrtens$ pip list
Package    Version
---------- -------
numpy      1.24.2
pip        23.0.1
setuptools 65.5.0

I have cloned the pyOptSparse repo onto my computer, navigated into the root folder of the repo, and executed pip install -e .. This is what it outputted:

Build Output

(openmdao_env) Matthews-MacBook-Pro:pyoptsparse mcmehrtens$ pip install -e .
Obtaining file:///Users/mcmehrtens/Documents/Iowa%20State%20University/Research/Libraries/pyoptsparse
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... error
  error: subprocess-exited-with-error
  
  × Getting requirements to build editable did not run successfully.
  │ exit code: 1
  ╰─> [22 lines of output]
      Traceback (most recent call last):
        File "/Users/mcmehrtens/pyenv/openmdao_env/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/Users/mcmehrtens/pyenv/openmdao_env/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/mcmehrtens/pyenv/openmdao_env/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 132, in get_requires_for_build_editable
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/2c/q_0zfh1526bb7wzr38lt835c0000gn/T/pip-build-env-p0v4hcf1/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 447, in get_requires_for_build_editable
          return self.get_requires_for_build_wheel(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/2c/q_0zfh1526bb7wzr38lt835c0000gn/T/pip-build-env-p0v4hcf1/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 338, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/2c/q_0zfh1526bb7wzr38lt835c0000gn/T/pip-build-env-p0v4hcf1/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 320, in _get_build_requires
          self.run_setup()
        File "/private/var/folders/2c/q_0zfh1526bb7wzr38lt835c0000gn/T/pip-build-env-p0v4hcf1/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 335, in run_setup
          exec(code, locals())
        File "<string>", line 81, in <module>
        File "<string>", line 35, in run_meson_build
      FileNotFoundError: [Errno 2] No such file or directory: 'meson_build/setup.log'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build editable did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

I am concerned about this line from the installation instructions:

We do not support operating systems other than Linux.

Is this my dead-end? I attempted to use Conda to perform the installation, but I ran into different build errors. I would appreciate any and all support on this issue :)

Edit:

Here is is the output from when I attempt to perform the build using the Conda source build instructions:

Conda Source Build

(openmdao_env) Matthews-MacBook-Pro:pyoptsparse mcmehrtens$ conda create -y -n pyos-build
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /Users/mcmehrtens/anaconda3/envs/pyos-build



Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate pyos-build
#
# To deactivate an active environment, use
#
#     $ conda deactivate

(openmdao_env) Matthews-MacBook-Pro:pyoptsparse mcmehrtens$ conda activate pyos-build
(pyos-build) (openmdao_env) Matthews-MacBook-Pro:pyoptsparse mcmehrtens$ conda config --env --add channels conda-forge
(pyos-build) (openmdao_env) Matthews-MacBook-Pro:pyoptsparse mcmehrtens$ conda config --env --set channel_priority strict
(pyos-build) (openmdao_env) Matthews-MacBook-Pro:pyoptsparse mcmehrtens$ conda env update -f .github/environment.yml
Collecting package metadata (repodata.json): done
Solving environment: done

Downloading and Extracting Packages

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate /Users/mcmehrtens/anaconda3/envs/pyos-build
#
# To deactivate an active environment, use
#
#     $ conda deactivate

(pyos-build) (openmdao_env) Matthews-MacBook-Pro:pyoptsparse mcmehrtens$ export IPOPT_DIR="$CONDA_PREFIX"
(pyos-build) (openmdao_env) Matthews-MacBook-Pro:pyoptsparse mcmehrtens$ python -m build -n -x .
* Building sdist...
Traceback (most recent call last):
  File "/Users/mcmehrtens/anaconda3/envs/pyos-build/lib/python3.11/site-packages/pep517/in_process/_in_process.py", line 351, in <module>
    main()
  File "/Users/mcmehrtens/anaconda3/envs/pyos-build/lib/python3.11/site-packages/pep517/in_process/_in_process.py", line 333, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mcmehrtens/anaconda3/envs/pyos-build/lib/python3.11/site-packages/pep517/in_process/_in_process.py", line 302, in build_sdist
    return backend.build_sdist(sdist_directory, config_settings)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mcmehrtens/anaconda3/envs/pyos-build/lib/python3.11/site-packages/setuptools/build_meta.py", line 417, in build_sdist
    return self._build_with_temp_dir(['sdist', '--formats', 'gztar'],
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mcmehrtens/anaconda3/envs/pyos-build/lib/python3.11/site-packages/setuptools/build_meta.py", line 398, in _build_with_temp_dir
    self.run_setup()
  File "/Users/mcmehrtens/anaconda3/envs/pyos-build/lib/python3.11/site-packages/setuptools/build_meta.py", line 335, in run_setup
    exec(code, locals())
  File "<string>", line 81, in <module>
  File "<string>", line 35, in run_meson_build
FileNotFoundError: [Errno 2] No such file or directory: 'meson_build/setup.log'

ERROR Backend subproccess exited when trying to invoke build_sdist

When reproducing my problem, I accidentally had both my Conda virtual environment and my normal venv activated at the same time, but regardless, it produced the same build error.

[whophil]

@mcmehrtens If you are OK using conda, you should be able to install pyoptsparse from conda-forge directly.

conda install -c conda-forge pyoptsparse

conda-forge has pre-built packages for both osx64 and osx-arm64 platforms.

Combining that with your own SNOPT module will take a little more doing, however. I just opened this #337 and #338 to help with that, but it's only half of the puzzle - you'll still need to build your own binding module.

In case you're not aware, it's possible to use x64 conda installation on Mac ARM via Rosetta. If you do that, any conda install command will install x86 packages instead of arm64. This can be useful as many legacy packages are not yet ready for arm64.

[pollinico]

Hi, thank you for your comments. I am having similar problems.
I compiled SNOPT on my macos with M1 chip.
How do you create a SNOPT module ? I have no idea, can you please give some directions?
I would try to install pyOptSparse from conda and link it to my compiled SNOPT.
Thank you!

[marcomangano]

Hello Nicolò, have you checked this section of the documentation for more details? I am not familiar with local SNOPT compilation, but you should have obtained a binding file that can be linked to pyOptSparse with the instructions in the link.

[whophil]

Hi @pollinico - assuming you already have libsnopt7.dylib, something like this should work:

# run this in a directory which contains `libsnopt7.dylib`

# fetch pyoptsparse source so we have the f2py and .f files
git clone https://github.com/mdolab/pyoptsparse.git

# compile f2py module
# you must have python available with numpy installed and a fortran compiler available on path
SRC=pyoptsparse/pyoptsparse/pySNOPT
python -m numpy.f2py -m snopt -c ${SRC}/f2py/snopt.pyf ${SRC}/*.f -L${PREFIX}/lib -lsnopt7

This should produce snopt.dylib which links libsnopt7.dylib dynamically. This means you need both .dylib files for SNOPT to work.

Then you can follow the instructions in the docs, setting the environment variable:

PYOPTSPARSE_IMPORT_SNOPT_FROM=/path/where/you/compiled/snopt.so