Difficulty building charm4py with mpi on ubuntu

[doncandela]

I'm trying to build charm4py with mpi on an Ubuntu 20.04 system, get an error failure I don't know how to interpret, any suggestions will be appreciated. Here's the output:

(p37m) dc:~$ pip3 install --install-option="--mpi" charm4py --no-binary charm4py
/home/dc/anaconda3/envs/p37m/lib/python3.7/site-packages/pip/_internal/commands/install.py:234: UserWarning: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option.
  cmdoptions.check_install_build_global(options)
Collecting charm4py
  Using cached charm4py-1.0.tar.gz (3.4 MB)
Requirement already satisfied: numpy>=1.10.0 in ./anaconda3/envs/p37m/lib/python3.7/site-packages (from charm4py) (1.19.2)
Requirement already satisfied: greenlet in ./anaconda3/envs/p37m/lib/python3.7/site-packages (from charm4py) (0.4.17)
Skipping wheel build for charm4py, due to binaries being disabled for it.
Installing collected packages: charm4py
    Running setup.py install for charm4py ... error
    ERROR: Command errored out with exit status 1:
     command: /home/dc/anaconda3/envs/p37m/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-3t3sz47v/charm4py_4d1d5183927c4c56bbcfc59e63ca5f58/setup.py'"'"'; __file__='"'"'/tmp/pip-install-3t3sz47v/charm4py_4d1d5183927c4c56bbcfc59e63ca5f58/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-eam4l2tn/install-record.txt --single-version-externally-managed --compile --install-headers /home/dc/anaconda3/envs/p37m/include/python3.7m/charm4py --mpi
         cwd: /tmp/pip-install-3t3sz47v/charm4py_4d1d5183927c4c56bbcfc59e63ca5f58/
    Complete output (129 lines):
    fatal: not a git repository (or any of the parent directories): .git
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.7
    creating build/lib.linux-x86_64-3.7/charm4py
    creating build/lib.linux-x86_64-3.7/charm4py/.libs
    creating /tmp/pip-install-3t3sz47v/charm4py_4d1d5183927c4c56bbcfc59e63ca5f58/charm4py/.libs
    creating build/lib.linux-x86_64-3.7/charmrun
    Uncompressing charm.tar.gz...
    Creating dir: mpi-linux-x86_64
    Creating dir: mpi-linux-x86_64/tmp
    Soft-linking over bin
    Soft-linking over lib
    Soft-linking over lib_so
    Soft-linking over include
    Soft-linking over tmp
    Copying src/scripts/Makefile to mpi-linux-x86_64/tmp
    Generating mpi-linux-x86_64/tmp/conv-mach-pre.sh
    Generating mpi-linux-x86_64/tmp/conv-mach-opt.h, conv-mach-opt.sh, conv-mach-opt.mak
    Performing 'make -j 2 basics OPTS=-optimize -production QUIET= CONFIG_OPTS=--disable-controlpoint --disable-tracing --disable-tracing-commthread --disable-charmdebug --disable-replay --disable-error-checking --disable-stats --enable-charmpy' in mpi-linux-x86_64/tmp
    pwd
    /tmp/pip-install-3t3sz47v/charm4py_4d1d5183927c4c56bbcfc59e63ca5f58/charm_src/charm/mpi-linux-x86_64/tmp
    find . -type l -exec rm {} \;
    rm -rf QuickThreads
    rm -rf hwloc
    rm -rf libs
    rm -rf ../bin ; mkdir ../bin
    if case mpi-linux-x86_64 in *win64*) true;; *-win-*) true;; *) false;; esac; then \
    	cp -f ../../src/arch/win/unix2nt* ../bin; \
    	cp -f ../../src/arch/win/createlink.exe ../bin; \
    fi
    rm -rf ../lib ; mkdir ../lib
    rm -rf ../lib_so ; mkdir ../lib_so; touch ../lib_so/.charmso
    rm -rf ../benchmarks
    rm -rf ../examples
    rm -rf ../tests
    ../../src/scripts/gatherflat ../../src/scripts          .
    ./gatherflat ../../src/conv-core                        .
    ./gatherflat ../../src/conv-ldb                         .
    ./gatherflat ../../src/conv-ccs                         .
    ./gatherflat ../../src/conv-perf                        .
    ./gatherflat ../../src/ck-core                          .
    ./gatherflat ../../src/ck-perf                          .
    ./gatherflat ../../src/ck-pics                          .
    ./gatherflat ../../src/ck-ldb                           .
    ./gatherflat ../../src/ck-cp                            .
    ./gatherflat ../../src/langs/simplemsg                  .
    ./gatherflat ../../src/langs/pvmc                       .
    ./gatherflat ../../src/langs/bluegene                   .
    ./gatherflat ../../src/langs/f90charm                   .
    ./gatherflat ../../src/xlatcpm                          .
    ./gathertree ../../src/QuickThreads                     QuickThreads
    ./gathertree ../../src/../contrib/hwloc                 hwloc
    ./gathertree ../../src/libs                             libs
    ./gathertree ../../src/arch/util                        .
    ./gathertree ../../src/arch/util/proc_management        proc_management
    ./gathertree ../../src/util                             .
    ./gathertree ../../src/langs                            langs
    ./gathertree ../../src/langs/jade                       langs/jade
    ./gathertree ../../src/arch/common                      .
    ./gathertree ../../src/arch/mpi                 .
    test -f ../../src/arch/mpi/gdir_link && cat ../../src/arch/mpi/gdir_link > .gdir.new && ./gathertree ../../src/arch/`cat .gdir.new`      . || true
    ./gatherflat ../../src/arch/mpi-linux-x86_64                 .
    ./gathertree ../../src/../benchmarks                    ../benchmarks
    ./gathertree ../../src/../examples                      ../examples
    ./gathertree ../../src/../tests                         ../tests
    rm -f ../bin/dep.pl ; cp dep.pl ../bin/
    chmod +x charmc
    ./system_ln  ../tmp/charmc ../bin/
    ./system_ln  ../../src/scripts/testrun ../bin/
    ./system_ln  ../tmp/tree.txt ../bin/
    ./system_ln  ../tmp/fuzzytree.txt ../bin/
    rm -rf ../include ; mkdir ../include
    ./system_ln  ../tmp/conv-*.*h ../include
    ./system_ln  ../tmp/cc-*.*h ../include
    ./system_ln  ../tmp/conv-mach-opt.sh ../include
    ./system_ln  ../tmp/conv-mach-opt.mak ../include
    if [ -x ./special.sh ] ; then SRCBASE=../../src ./special.sh ; fi
    if [ ! -f conv-common.h ] ; then ( touch conv-common.h ) ; fi
    touch dirs+sources
    make -C topomanager -f Makefile_charm headers
    make[1]: Entering directory '/tmp/pip-install-3t3sz47v/charm4py_4d1d5183927c4c56bbcfc59e63ca5f58/charm_src/charm/mpi-linux-x86_64/tmp/topomanager'
    /bin/cp TopoManager.h ../../include
    /bin/cp BGQTorus.h ../../include
    /bin/cp XTTorus.h ../../include
    echo "//empty" > topomanager_config.h
    /bin/cp topomanager_config.h ../../include
    make[1]: Leaving directory '/tmp/pip-install-3t3sz47v/charm4py_4d1d5183927c4c56bbcfc59e63ca5f58/charm_src/charm/mpi-linux-x86_64/tmp/topomanager'
    make charmxi
    make[1]: Entering directory '/tmp/pip-install-3t3sz47v/charm4py_4d1d5183927c4c56bbcfc59e63ca5f58/charm_src/charm/mpi-linux-x86_64/tmp'
    autoreconf
    autoheader
    automake: warning: am_libhwloc_embedded_la_rpath was already defined in condition TRUE, which includes condition !HWLOC_BUILD_STANDALONE ...
    configure.ac:2972: ... 'am_libhwloc_embedded_la_rpath' previously defined here
    ./configure --disable-controlpoint --disable-tracing --disable-tracing-commthread --disable-charmdebug --disable-replay --disable-error-checking --disable-stats --enable-charmpy
    Error checking is disabled
    AMPI error checking is enabled
    Statistics collection is disabled
    Charm tracing is disabled
    Charm tracing communication thread is disabled
    Task Queue is disabled
    Drone mode is disabled
    CharmDebug is disabled
    Charm record/replay is disabled
    CCS is enabled
    Charm control point is disabled
    Charm LB user data is disabled
    Lockless queue for PE local and node queue is disabled
    Charm shrink expand is disabled
    charm4py support is enabled
    Setting load balancing timer type as 'double'
    checking machine name... mpi-linux-x86_64
    checking "cp command as"... cp -p
    checking "C++ compiler as"... "mpicxx   "
    checking "whether C++ compiler works"... "no"
    Cannot compile C++ programs with mpicxx
     (check your charm++ version)
    *** Please find detailed output in tmp/charmconfig.out ***
    make[1]: *** No rule to make target 'conv-autoconfig.h', needed by 'xi-main.o'.  Stop.
    make[1]: Leaving directory '/tmp/pip-install-3t3sz47v/charm4py_4d1d5183927c4c56bbcfc59e63ca5f58/charm_src/charm/mpi-linux-x86_64/tmp'
    make: *** [Makefile:331: headers] Error 2
    -------------------------------------------------
    Charm++ NOT BUILT. Either cd into mpi-linux-x86_64/tmp and try
    to resolve the problems yourself, visit
    http://charm.cs.illinois.edu/
    for more information. Otherwise, email the developers at charm@cs.illinois.edu
    error: An error occured while building charm library
    ----------------------------------------
ERROR: Command errored out with exit status 1: /home/dc/anaconda3/envs/p37m/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-3t3sz47v/charm4py_4d1d5183927c4c56bbcfc59e63ca5f58/setup.py'"'"'; __file__='"'"'/tmp/pip-install-3t3sz47v/charm4py_4d1d5183927c4c56bbcfc59e63ca5f58/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-eam4l2tn/install-record.txt --single-version-externally-managed --compile --install-headers /home/dc/anaconda3/envs/p37m/include/python3.7m/charm4py --mpi Check the logs for full command output.
(p37m) dc:~$

[matthiasdiener]

Can you paste the content of tmp/charmconfig.out ?

[doncandela]

After installing openmpi and openmpi-cxx the make gets quite a bit further, generating 980 lines of output but then giving "Fatal Error,,, No F90 compiler (CMK_CF90) defined". Installing gfortran did not help - do I need to install a different F90 compiler? Here is the output from the point of error onwards:

   ../bin/charmc  -optimize -production  -build-shared -o tracef_f.o -c tracef_f.f90 && ../bin/charmc  -optimize -production  -build-shared -cpmod ../include tracemod.M  || touch tracef_f.o
    Fatal Error by charmc in directory /tmp/pip-install-1rvd2c41/charm4py_68f1be5847de4bbba3b53859735b7149/charm_src/charm/mpi-linux-x86_64/tmp
       No F90 compiler (CMK_CF90) defined
    charmc exiting...
    ../bin/charmc  -optimize -production  -build-shared -I. -o libmemory-default.o -DCMK_MEMORY_BUILD_DEFAULT memory.C
    ../bin/charmc  -optimize -production  -build-shared -I. -o libmemory-os.o -DCMK_MEMORY_BUILD_OS memory.C
    ../bin/charmc  -optimize -production  -build-shared -I. -o ../lib/libmemory-default.a -DCMK_MEMORY_BUILD_DEFAULT libmemory-default.o
    ar: creating ../lib/libmemory-default.a
    ../bin/charmc  -optimize -production  -build-shared -I. -o libmemory-os-verbose.o     -DCMK_MEMORY_BUILD_OS_WRAPPED -DCMK_MEMORY_BUILD_VERBOSE    -touch-on-failure memory.C
    ../bin/charmc  -optimize -production  -build-shared -I. -o ../lib/libmemory-os.a -DCMK_MEMORY_BUILD_OS libmemory-os.o
    memory.C: In function 'void* aligned_alloc(size_t, size_t)':
    memory.C:388:16: error: redefinition of 'void* aligned_alloc(size_t, size_t)'
     CLINKAGE void *aligned_alloc(size_t align, size_t size) CMK_THROW { return meta_aligned_alloc(align,size); }
                    ^~~~~~~~~~~~~
    In file included from /home/dc/anaconda3/envs/p37m/x86_64-conda_cos6-linux-gnu/include/c++/7.3.0/cstdlib:75:0,
                     from /home/dc/anaconda3/envs/p37m/x86_64-conda_cos6-linux-gnu/include/c++/7.3.0/stdlib.h:36,
                     from memory.C:54:
    /home/dc/anaconda3/envs/p37m/x86_64-conda_cos6-linux-gnu/sysroot/usr/include/stdlib.h:513:21: note: 'void* aligned_alloc(size_t, size_t)' previously defined here
     static inline void* aligned_alloc (size_t al, size_t sz)
                         ^~~~~~~~~~~~~
    ../bin/charmc  -optimize -production  -build-shared -I. -o ../lib/libmemory-os-verbose.a     -DCMK_MEMORY_BUILD_OS_WRAPPED -DCMK_MEMORY_BUILD_VERBOSE    -touch-on-failure libmemory-os-verbose.o
    ar: creating ../lib/libmemory-os.a
    ../bin/charmc  -optimize -production  -build-shared -I. -o libmemory-os-isomalloc.o   -DCMK_MEMORY_BUILD_OS_WRAPPED -DCMK_MEMORY_BUILD_ISOMALLOC  -touch-on-failure memory.C
    Fatal Error by charmc in directory /tmp/pip-install-1rvd2c41/charm4py_68f1be5847de4bbba3b53859735b7149/charm_src/charm/mpi-linux-x86_64/tmp
       libmemory-os-verbose.o: file not recognized: File truncated
    charmc exiting...
    make: *** [Makefile:672: ../lib/libmemory-os-verbose.a] Error 1
    make: *** Waiting for unfinished jobs....
    memory.C: In function 'void* aligned_alloc(size_t, size_t)':
    memory.C:388:16: error: redefinition of 'void* aligned_alloc(size_t, size_t)'
     CLINKAGE void *aligned_alloc(size_t align, size_t size) CMK_THROW { return meta_aligned_alloc(align,size); }
                    ^~~~~~~~~~~~~
    In file included from /home/dc/anaconda3/envs/p37m/x86_64-conda_cos6-linux-gnu/include/c++/7.3.0/cstdlib:75:0,
                     from /home/dc/anaconda3/envs/p37m/x86_64-conda_cos6-linux-gnu/include/c++/7.3.0/stdlib.h:36,
                     from memory.C:54:
    /home/dc/anaconda3/envs/p37m/x86_64-conda_cos6-linux-gnu/sysroot/usr/include/stdlib.h:513:21: note: 'void* aligned_alloc(size_t, size_t)' previously defined here
     static inline void* aligned_alloc (size_t al, size_t sz)
                         ^~~~~~~~~~~~~
    ../bin/charmc  -optimize -production  -build-shared -I. -o ../lib/libmemory-os-isomalloc.a   -DCMK_MEMORY_BUILD_OS_WRAPPED -DCMK_MEMORY_BUILD_ISOMALLOC  -touch-on-failure libmemory-os-isomalloc.o
    Fatal Error by charmc in directory /tmp/pip-install-1rvd2c41/charm4py_68f1be5847de4bbba3b53859735b7149/charm_src/charm/mpi-linux-x86_64/tmp
       libmemory-os-isomalloc.o: file not recognized: File truncated
    charmc exiting...
    make: *** [Makefile:675: ../lib/libmemory-os-isomalloc.a] Error 1
    -------------------------------------------------
    Charm++ NOT BUILT. Either cd into mpi-linux-x86_64/tmp and try
    to resolve the problems yourself, visit
    http://charm.cs.illinois.edu/
    for more information. Otherwise, email the developers at charm@cs.illinois.edu
    error: An error occured while building charm library
    ----------------------------------------
ERROR: Command errored out with exit status 1: /home/dc/anaconda3/envs/p37m/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-1rvd2c41/charm4py_68f1be5847de4bbba3b53859735b7149/setup.py'"'"'; __file__='"'"'/tmp/pip-install-1rvd2c41/charm4py_68f1be5847de4bbba3b53859735b7149/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-vakzv6ye/install-record.txt --single-version-externally-managed --compile --install-headers /home/dc/anaconda3/envs/p37m/include/python3.7m/charm4py --mpi Check the logs for full command output.
(p37m) dc:~$ 

[matthiasdiener]

The Fortran error is ignored by the build system, and should not affect your build (unless you intend to use Charm++'s Fortran bindings). The reason the build fails is due to this error:

memory.C: In function 'void* aligned_alloc(size_t, size_t)':
    memory.C:388:16: error: redefinition of 'void* aligned_alloc(size_t, size_t)'
     CLINKAGE void *aligned_alloc(size_t align, size_t size) CMK_THROW { return meta_aligned_alloc(align,size); }
                    ^~~~~~~~~~~~~
    In file included from /home/dc/anaconda3/envs/p37m/x86_64-conda_cos6-linux-gnu/include/c++/7.3.0/cstdlib:75:0,
                     from /home/dc/anaconda3/envs/p37m/x86_64-conda_cos6-linux-gnu/include/c++/7.3.0/stdlib.h:36,
                     from memory.C:54:
    /home/dc/anaconda3/envs/p37m/x86_64-conda_cos6-linux-gnu/sysroot/usr/include/stdlib.h:513:21: note: 'void* aligned_alloc(size_t, size_t)' previously defined here
     static inline void* aligned_alloc (size_t al, size_t sz)
                         ^~~~~~~~~~~~~

Maybe @evan-charmworks has a better idea what the issue is. @doncandela Can you specify your Charm++ version, and your ./build command?

[doncandela]

The only command I issued was

$ pip3 install --install-option="--mpi" charm4py --no-binary charm4py

The output is about 1000 lines, I have pasted it into this text file:
2021-05-16_pip_output.txt

[doncandela]

I have resolved this issue an successfully built charm4py with mpi.
In case it might be useful for someone else, I think the problem was that I had two competing C++'s installed. Installing openmpi-mpicxx on anaconda also installed gxx, but this was not recognized by the charm setup which required me to install g++ (sudo apt install g++). The solution was to not install mpi on anaconda, but instead from the terminal (sudo apt-get install openmpi-bin libopenmpi-dev).

[matthiasdiener]

Thank you for following up @doncandela!