macos M3

[man74cio]

I am trying to run a program that uses Pyccel on a macOS (m3 processor). It has always run well on Linux and benefited from functions compiled with Pyccel. Now the program seems significantly slower (at the speed of regular Python). Is there any particular reason for this?

the command :

pyccel  ssc_model/my_sum.py  --openmp

gives me:

py3env/lib/python3.12/site-packages/pyccel/codegen/compiling/compilers.py:502: UserWarning: ld: warning: ignoring duplicate libraries: '-lgomp'
  warnings.warn(UserWarning(err))

[EmilyBourne]

Hello,

I am not sure why this would be the case. @yguclu has more experience with MacOS than me. Maybe he has an idea?

at the speed of regular Python

The first thing that comes to mind when I see this is that maybe your system is not picking up the file. You seem to be compiling for Python 3.12 is that the version you are using to run?

The easiest way to see if the file is being used or not is to modify it and see if the modification shows up in Python. The steps are:

1. Use pyccel ssc_model/my_sum.py --openmp --verbose to see the compilation commands that we are using.
2. Open ssc_model/__pyccel__/my_sum.f90 and add a print function to the code (e.g. print *, "hello")
3. Recompile the .so file using the various compiler commands printed in verbose mode
4. Move the newly generated .so file from ssc_model/__pyccel__/ to ``ssc_model`
5. Rerun your code and check that the print appears

If it doesn't then there is probably a problem with the ABI version (e.g. different Python version, different system, different Python interpreter).

Please give this a go and let us know if the translated file is being used

[yguclu]

Hi @man74cio, it is great to hear that Pyccel is being useful to your work!

The procedure suggested by @EmilyBourne can be further simplified as a first step. After calling Pyccel on your Python file my_sum.py a shared library my_sum.[python-version-info].so is created in the same folder. I suggest that you temporarily move the Python file to a different location, so that your Python interpreter has to use the shared library just created.

Please try to run your code again and see what happens. Does it run at all? Is it fast or slow?

[man74cio]

Hello to all,
effectively it uses the my_sum.___.so library : it tested .

The compilation output is :

 pyccel  ssc_model/my_sum.py  --openmp --verbose
>>> treating :: /Users/mmancini/Progetti_vari/emblem_paloma/py3env/lib/python3.12/site-packages/pyccel/stdlib/internal/openmp.pyh
>>> treating :: /Users/mmancini/Progetti_vari/emblem_paloma/py3env/lib/python3.12/site-packages/pyccel/stdlib/internal/openmp.pyh
/opt/homebrew/bin/gfortran -O3 -funroll-loops -fPIC -fopenmp -c -I /Users/mmancini/Progetti_vari/emblem_paloma/ssc_model/__pyccel__ /Users/mmancini/Progetti_vari/emblem_paloma/ssc_model/__pyccel__/my_sum.f90 -o /Users/mmancini/Progetti_vari/emblem_paloma/ssc_model/__pyccel__/my_sum.o -J /Users/mmancini/Progetti_vari/emblem_paloma/ssc_model/__pyccel__
/opt/homebrew/bin/gfortran -O3 -funroll-loops -fPIC -fopenmp -c -I /Users/mmancini/Progetti_vari/emblem_paloma/ssc_model/__pyccel__ /Users/mmancini/Progetti_vari/emblem_paloma/ssc_model/__pyccel__/bind_c_my_sum.f90 -o /Users/mmancini/Progetti_vari/emblem_paloma/ssc_model/__pyccel__/bind_c_my_sum.o -J /Users/mmancini/Progetti_vari/emblem_paloma/ssc_model/__pyccel__
/usr/bin/gcc -O3 -funroll-loops -fPIC -fno-strict-overflow -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk -Xpreprocessor -fopenmp -c -I /Users/mmancini/Progetti_vari/emblem_paloma/ssc_model/__pyccel__/ndarrays -I /Users/mmancini/Progetti_vari/emblem_paloma/ssc_model/__pyccel__ -I /Users/mmancini/Progetti_vari/emblem_paloma/ssc_model/__pyccel__/cwrapper -I /Users/mmancini/Progetti_vari/emblem_paloma/ssc_model/__pyccel__/cwrapper_ndarrays -I /opt/homebrew/opt/python@3.12/Frameworks/Python.framework/Versions/3.12/include/python3.12 -I /Users/mmancini/Progetti_vari/emblem_paloma/py3env/lib/python3.12/site-packages/numpy/core/include -I /usr/local/opt/libomp/include /Users/mmancini/Progetti_vari/emblem_paloma/ssc_model/__pyccel__/my_sum_wrapper.c -o /Users/mmancini/Progetti_vari/emblem_paloma/ssc_model/__pyccel__/my_sum_wrapper.o
/opt/homebrew/bin/gfortran -shared -O3 -funroll-loops -fPIC -fopenmp -I /Users/mmancini/Progetti_vari/emblem_paloma/ssc_model/__pyccel__/ndarrays -I /Users/mmancini/Progetti_vari/emblem_paloma/ssc_model/__pyccel__ -I /Users/mmancini/Progetti_vari/emblem_paloma/ssc_model/__pyccel__/cwrapper -I /Users/mmancini/Progetti_vari/emblem_paloma/ssc_model/__pyccel__/cwrapper_ndarrays -I /opt/homebrew/opt/python@3.12/Frameworks/Python.framework/Versions/3.12/include/python3.12 -I /Users/mmancini/Progetti_vari/emblem_paloma/py3env/lib/python3.12/site-packages/numpy/core/include /Users/mmancini/Progetti_vari/emblem_paloma/ssc_model/__pyccel__/my_sum_wrapper.o /Users/mmancini/Progetti_vari/emblem_paloma/ssc_model/__pyccel__/bind_c_my_sum.o /Users/mmancini/Progetti_vari/emblem_paloma/ssc_model/__pyccel__/ndarrays/ndarrays.o /Users/mmancini/Progetti_vari/emblem_paloma/ssc_model/__pyccel__/my_sum.o /Users/mmancini/Progetti_vari/emblem_paloma/ssc_model/__pyccel__/cwrapper/cwrapper.o /Users/mmancini/Progetti_vari/emblem_paloma/ssc_model/__pyccel__/cwrapper_ndarrays/cwrapper_ndarrays.o /opt/homebrew/opt/python@3.12/Frameworks/Python.framework/Versions/3.12/lib/libpython3.12.dylib -o /Users/mmancini/Progetti_vari/emblem_paloma/ssc_model/__pyccel__/my_sum.cpython-312-darwin.so -lgomp
/Users/mmancini/Progetti_vari/emblem_paloma/py3env/lib/python3.12/site-packages/pyccel/codegen/compiling/compilers.py:502: UserWarning: ld: warning: ignoring duplicate libraries: '-lgomp'

  warnings.warn(UserWarning(err))
> Shared library has been created: /Users/mmancini/Progetti_vari/emblem_paloma/ssc_model/my_sum.cpython-312-darwin.so

[EmilyBourne]

effectively it uses the my_sum.___.so library : it tested .

Je ne suis pas sûre d'avoir bien compris c'est bien en train d'utiliser la librarie?

[EmilyBourne]

What are you using for your tests? I see that your file is called my_sum.py does it just implement a sum function? If so then I would not necessarily expect it to be faster than NumPy as they also use OpenMP. Are you comparing the times for the same file (in Python and Fortran)?

We could also try and see if OpenMP is indeed running by setting the environment variable OMP_NUM_THREADS if the code gets slower when fewer threads are used then the OpenMP paradigms must be working

[man74cio]

my_sum is only the name of the first example I used and it contains mainly two functions:
my_simps : which replace de simps of numpy
my_I_ic : which is a more complex function which contains a omp for.

On my workstation (10 cores) the speedup was impressive.

I put here a brief comparison between the m3-macos and my older laptop (2 cores).
M3 : without pyccel : 11.3 s
M3 : with. pyccel : 7.7 s
INTEL : without pyceel : 4.9 s
INTEL: with pycell : 0.78 s

[EmilyBourne]

How many cores does your m3 have? Have you tried changing OMP_NUM_THREADS to see if it's using them?

[EmilyBourne]

It is interesting that the version without Pyccel is also slower on your new computer. Are you using the same versions of Python on the two machines?

[man74cio]

Hello,
this morning I tried it with the laptop plugged in and changing the number of threads but the result is the same. In any case you should now that the openmp part is not very important for the speedup.

[EmilyBourne]

By the result is the same do you mean that OpenMP is not working? Maybe we are using the wrong flags (or OpenMP version) for your setup. What compiler version are you using?

[man74cio]

OpenMP is working but it is not very useful at the moment because the parallel loop is within a function that, in turn, is called many times. I need to move it outside the function.
Concerning the python version: in my old laptop python3.8 , in the macos : 3.12
The gfortran version is : GNU Fortran (Homebrew GCC 13.2.0) 13.2.0

[EmilyBourne]

In that case I am not sure what is slowing it down. Python 3.12 is usually faster than Python 3.8 (a lot of the developments in 3.11 and 3.12 were performance based). Your compiler also seems pretty recent. Maybe you have a slow version of Python installed (see https://pythonspeed.com/articles/faster-python/ for a discussion on how the build version of Python can affect performance). Pyccel uses the same compilation flags that were used to compile Python so any poor flag choices could also affect the Pyccel performance.

Is is possible that you may be using different versions of Pyccel on the two machines? We made some changes to the C-Python interface last year (v1.8.0) to allow slices to be passed to the translated functions. This will have slowed down the interface a little (see discussion here: #1608). Given how slow loops are in Python we usually assume that the loops are inside the translated function. When this is the case performance issues in the interface should be negligible. Here it sounds like your loop is around the Pyccelised function. In that case any performance issues in the wrapper would be amplified. That said I don't think the changes should have a huge impact unless you have lots of optional arguments (e.g. a = None)

[man74cio]

Hello I tried to re-nstall python from the site and all the dependances but the result does not change.
I also tryed to compile numpy but it does not work at all :
pip install numpy --force-reinstall --no-deps --no-cache --no-binary :all: --compile

[man74cio]

Hello Emily,
after some problem with gfortran and gcc I could reinstall correctly them and now the times are fairly better now.
M3 : without pyccel : ~6 s
M3 : with. pyccel : ~2.2 s

I don't understand completely why this behavior