openmp support

[lukeolson]

As a start:
https://github.com/lukeolson/sparse-ops

Steps:

1. add SpMV kernels for CSR, BSR, and blocked vectors
2. add SpMM kernels
3. modify setup to check for -fopenmp
4. profile amg_core for next kernels

[ma-sadeghi]

@lukeolson Can I ask what the status of the OpenMP support is? Thanks!

[jbschroder]

Hi Amin, The OpenMP support is currently still in development, unfortunately. We have some working prototypes (in particular from Luke) on adding threaded parallelism, but they aren't ready for release yet. Best, Jacob

…

On Thu, Nov 8, 2018 at 2:04 PM Amin Sadeghi ***@***.***> wrote: @lukeolson <https://github.com/lukeolson> Can I ask what the status of the OpenMP support is? Thanks! — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#227 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEjD47Hl9exE0czWPTD89Xsevr42Qgwtks5utJxpgaJpZM4VJwxb> .

[lukeolson]

I think we're getting close.
https://github.com/lukeolson/sparse-ops

It will take some testing to make sure the install is polished.

[jbschroder]

That's pretty cool. I hadn't seen the plots yet.

…

On Thu, Nov 8, 2018 at 2:35 PM Luke Olson ***@***.***> wrote: I think we're getting close. https://github.com/lukeolson/sparse-ops It will take some testing to make sure the install is polished. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#227 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEjD45yGVq7bWH1AYehMqrx3n19iUKmOks5utKOigaJpZM4VJwxb> .

[ma-sadeghi]

Hi @lukeolson,

Do you think it's possible that I somehow use what you've been working so far on OpenMP support, before it's included it in official PyAMG? Maybe a sample script (some sort of hack) to leverage the ongoing work. I'm desperately in need of a fast solver as I need to do some sensitivity analysis for quite a few parameters in a relatively huge system (100 to 1000 million unknowns).

I know you're probably busy, and I really appreciate your efforts.

Cheers,
Amin

[lukeolson]

A first pass is here:
https://github.com/pyamg/pyamg/tree/omp

And in particular:
https://github.com/pyamg/pyamg/blob/omp/pyamg/util/sparse.py

This introduces an OMP SPMV:
https://github.com/pyamg/pyamg/blob/omp/pyamg/amg_core/sparse.h#L28

However I have a dimension mismatch in the test:
https://github.com/pyamg/pyamg/blob/omp/pyamg/util/tests/test_sparse.py

That I haven't been able to track down yet.

The build is fragile but it's serviceable. Sorry we haven't made any more progress at the moment.

[lukeolson]

Quick update. The omp branch works, but the openmp build flags are somewhat hardcoded. Working on that.

[ma-sadeghi]

@lukeolson Thanks for getting back to me. When I install the package from source, import pyamg throws this error

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/amin/Code/Repos/pyamg/pyamg/__init__.py", line 12, in <module>
    from .classical import ruge_stuben_solver
  File "/home/amin/Code/Repos/pyamg/pyamg/classical/__init__.py", line 4, in <module>
    from .classical import *
  File "/home/amin/Code/Repos/pyamg/pyamg/classical/classical.py", line 9, in <module>
    from pyamg.relaxation.smoothing import change_smoothers
  File "/home/amin/Code/Repos/pyamg/pyamg/relaxation/smoothing.py", line 5, in <module>
    from . import relaxation
  File "/home/amin/Code/Repos/pyamg/pyamg/relaxation/relaxation.py", line 10, in <module>
    from pyamg.util.utils import type_prep, get_diagonal, get_block_diag
  File "/home/amin/Code/Repos/pyamg/pyamg/util/__init__.py", line 7, in <module>
    from .utils import *
  File "/home/amin/Code/Repos/pyamg/pyamg/util/utils.py", line 13, in <module>
    import pyamg.amg_core
  File "/home/amin/Code/Repos/pyamg/pyamg/amg_core/__init__.py", line 5, in <module>
    from .evolution_strength import *
ModuleNotFoundError: No module named 'pyamg.amg_core.evolution_strength'

(Ubuntu 16.04, using Anaconda3's python, pyamg's master branch).
As for the omp branch, python setup.py build throws the following error:

(pmeal) amin@tinyflask:~$ python setup.py build
7be986fd98e0bd37e9d0c015e67072f12ea78d6c
4.0.0
running build
running build_py
copying pyamg/version.py -> build/lib.linux-x86_64-3.6/pyamg
copying pyamg/multilevel.py -> build/lib.linux-x86_64-3.6/pyamg
copying pyamg/amg_core/__init__.py -> build/lib.linux-x86_64-3.6/pyamg/amg_core
copying pyamg/util/__init__.py -> build/lib.linux-x86_64-3.6/pyamg/util
copying pyamg/util/sparse.py -> build/lib.linux-x86_64-3.6/pyamg/util
copying pyamg/util/tests/test_sparse.py -> build/lib.linux-x86_64-3.6/pyamg/util/tests
running build_ext
gcc -pthread -B /home/amin/anaconda3/envs/pmeal/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/amin/anaconda3/envs/pmeal/include/python3.6m -I/home/amin/anaconda3/envs/pmeal/lib/python3.6/site-packages/numpy/core/include -c /tmp/tmp3bz6yszf.cpp -o tmp/tmp3bz6yszf.o -std=c++14
gcc -pthread -B /home/amin/anaconda3/envs/pmeal/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/amin/anaconda3/envs/pmeal/include/python3.6m -I/home/amin/anaconda3/envs/pmeal/lib/python3.6/site-packages/numpy/core/include -c /tmp/tmp3ay83b56.cpp -o tmp/tmp3ay83b56.o -fopenmp
-fopenmp

OPENMP:  -fopenmp 
gcc -pthread -B /home/amin/anaconda3/envs/pmeal/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/amin/anaconda3/envs/pmeal/include/python3.6m -I/home/amin/anaconda3/envs/pmeal/lib/python3.6/site-packages/numpy/core/include -c /tmp/tmpdxmnjaf3.cpp -o tmp/tmpdxmnjaf3.o -fvisibility=hidden
building 'pyamg.amg_core.sparse' extension
gcc -pthread -B /home/amin/anaconda3/envs/pmeal/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/amin/anaconda3/envs/pmeal/include/python3.6m -I/home/amin/.local/include/python3.6m -I/home/amin/anaconda3/envs/pmeal/include/python3.6m -I/home/amin/anaconda3/envs/pmeal/lib/python3.6/site-packages/numpy/core/include -c pyamg/amg_core/sparse_bind.cpp -o build/temp.linux-x86_64-3.6/pyamg/amg_core/sparse_bind.o -DVERSION_INFO="4.0.0" -std=c++14 -fopenmp -I/usr/local/include -fvisibility=hidden
g++ -pthread -shared -B /home/amin/anaconda3/envs/pmeal/compiler_compat -L/home/amin/anaconda3/envs/pmeal/lib -Wl,-rpath=/home/amin/anaconda3/envs/pmeal/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.6/pyamg/amg_core/sparse_bind.o -o build/lib.linux-x86_64-3.6/pyamg/amg_core/sparse.cpython-36m-x86_64-linux-gnu.so -L/usr/local/lib 
g++: error: : No such file or directory
error: command 'g++' failed with exit status 1

Edit: probably the error when building the omp branch has something to do with what you said earlier?

but the openmp build flags are somewhat hardcoded

[isuruf]

Change the line at https://github.com/pyamg/pyamg/blob/omp/setup.py#L173 to return '-fopenmp', '-fopenmp' and it should work

[ma-sadeghi]

Thanks @isuruf, it works fine now. Another question: how do I change number of OMP threads? When I run try_omp.py under pyamg/scripts using my Core i5 Macbook, I almost get 2x speedup (as expected), but when I run it on my workstation which has 6 cores (12 threads), I still get almost 2x speedup.

[lukeolson]

OMP_NUM_THREADS=8 python3 try_omp.py

should work.

It's going to depend on the matrix. And this only impacts Av, Rv, and P*v in the solve phase. I also didn't doing an optimization of the pragma call -- it would be worth optimizing that. This was a first pass to get the structure in place. You may want to look at https://github.com/lukeolson/sparse-ops as well.

[ma-sadeghi]

It seems to somehow get overridden.

(pmeal) amin@tinyflask:~$ OMP_NUM_THREADS=12 python try_omp.py 
setup time 3.5868915810006
reg time 3.738760035999803
cg time 3.0501696480005194
omp time 1.7180426159993658

(pmeal) amin@tinyflask:~$ OMP_NUM_THREADS=6 python try_omp.py 
setup time 3.5990759360010998
reg time 3.7275303799997346
cg time 3.027595845000178
omp time 1.5286244850012736

(pmeal) amin@tinyflask:~$ OMP_NUM_THREADS=2 python try_omp.py 
setup time 3.6943484990006255
reg time 3.742394652001167
cg time 3.042061030999321
omp time 1.6502896710007917

(pmeal) amin@tinyflask:~$ OMP_NUM_THREADS=1 python try_omp.py 
setup time 3.5809689310008253
reg time 3.727625204001015
cg time 3.039052205000189
omp time 1.869501825000043

(cg time is scipy's cg solver)

[lukeolson]

I pushed a few fixes to omp (keep in mind the setup is not finalized). With this I added a small omp_info function.

python3 -c "import pyamg; pyamg.amg_core.omp_info()"

Output

Thread 0 of 8 total threads
Thread 4 of 8 total threads
Thread 1 of 8 total threads
Thread 7 of 8 total threads
Thread 3 of 8 total threads
Thread 2 of 8 total threads
Thread 6 of 8 total threads
Thread 5 of 8 total threads

OMP_NUM_THREADS=4 python3 -c "import pyamg; pyamg.amg_core.omp_info()"

output

Thread 0 of 4 total threads
Thread 1 of 4 total threads
Thread 2 of 4 total threads
Thread 3 of 4 total threads

[ma-sadeghi]

Do you think the omp branch can be merged into main? Thanks!