compile from source have some problems

[jamdodot]

Current Behavior

1. dependencies in cmake satge
   I still have this shellcheck not found ,and it is difficult to install shellcheck from source code . I wonder if it will affect the compilation if I don't install this
   

Steps to Reproduce (for bugs)

Please make sure to execute the reproduction steps with newly recreated and empty tmp folders.

make -j

1. build target mmseqs-framework
   
2. got error
   

Your Environment

Include as many relevant details about the environment you experienced the bug in.

• Which MetaEuk version was used (Statically-compiled, self-compiled, Homebrew, etc.):Metaeuk Release 6-a5d39d9
• For self-compiled and Homebrew: Compiler and Cmake versions used and their invocation: HUAWEI BiSheng Compiler 2.1.0.B010 clang version 12.0.0
• cmake version 3.12.1

If I want to fix make errors, do I start with cmake or do I modify the cpp source code?

[milot-mirdita]

You can ignore the shellcheck message. I would recommend to restrict the number of processes make launches.

Compiling with make -j 4 or so should work.

[jamdodot]

Yes！！！ make -j 4 can work，thank you very much！ but i still have dependency problem.

1. start to test this Binary files
   
2. missing shared library
   
3. search this shared library
   

It seems that the compiler is able to provide this file, but my attempts to import this file at the cmake stage failed

4. link error
   

I'm a little confused, do I need to install other software that provides libc++.so.1, or continue testing with this provided by the compiler

[milot-mirdita]

Can you please try to comment out lines 167-171 in this file:

metaeuk/lib/mmseqs/CMakeLists.txt

Line 167 in 1da320a

if (CMAKE_COMPILER_IS_CLANG AND (NOT EMSCRIPTEN))

And repeat the compilation with a clean cmake-build folder?

[jamdodot]

Still able to succeed ! Thank you for your enthusiasm and response. 👍 Also I need to learn the reason why commenting out these lines can work. 😃

[milot-mirdita]

Your compiler probably already links its own c++ std lib somehow. Not really sure whats going on.

I am quite interested in your use-case of Metaeuk on ARM.
How much RAM does your machine have? Metaeuk will probably struggle on a very low RAM machine.

[jamdodot]

I haven't started using Metaeuk yet， and the current server for building metaeuk is 4vCPUs 8GiB aarch64.

I would really like to test the functionality on ARM 😃

[jamdodot]

Sorry I have a problem with openMP 😭

1. problem

I use the following command to test metaeuk

*/metaeuk/build/bin/metaeuk easy-predict */test/test_metaeuk/two_contigs/contigs.fna */test/test_metaeuk/two_contigs/proteins.faa ressxp tmpsxp

and it shows

Calling program has OMP_PROC_BIND set in its environment. Please unset OMP_PROC_BIND.

these message can be find in this link

2.

I try to unset this Environment Variables

unset  OMP_PROC_BIND

But it didn't work.

Do i need to edit the CmakeLists?

[milot-mirdita]

You can disable the check here:

metaeuk/lib/mmseqs/src/commons/CommandCaller.cpp

Line 17 in 1da320a

#if _OPENMP >= 201307

[milot-mirdita]

If you see that the cpu utilization is stuck to a load of 1.0 instead of 4.0, then it’s probably some issue with the cpu affinity selection in the openmp implementation of your compiler.

[jamdodot]

Description

When I tested metaeuk, the arm machine I used had 128 cores, while the comparison x86 machine had only 8 (my laptop). At the same time, the number of files generated by the 128-core machine is much more than that of the x86. I wonder if it is because of some tasks that are repeatedly executed that the arm is time-consuming. If the number of cores is the same, is the execution time of the two machines similar? Why are there so many files generated on the arm server

Arm

x86

[milot-mirdita]

Metaeuk generates one output file per thread/core for intermediate result files. The final result files (fasta,gtf,etc.) should be merged and not be split.

[jamdodot]

Do you mean to execute all the contents of run.sh? In this case, arm is still much slower than x86, why is this?

[milot-mirdita]

run.sh is not very representative of MetaEuk's performance and just meant for quick sanity check in our continuous integration. It's much too tiny. I suspect ARM is spending most time in creating/tearing down threads.

[jamdodot]

It makes sense, does it have something to do with openMP? Is there a way to fix this threading problem? or use other independent tests to evaluate performanced 🤷🏻‍♂️

[milot-mirdita]

This is not a problem, this is expected behavior. To benchmark MetaEuk's runtime you need to run it on larger query/target sets.

[jamdodot]

Does this software use mpi for multi-process computing, or just use openMP for multi-thread computing

[jamdodot]

When using mpi to execute multiple processes, the two processes generate the same result file and cause an error to exit. How to solve this problem? can i use mpi for testing

[milot-mirdita]

If you use MPI, only the search stage will be parallelized. This is the most time consuming step usually.

You have to specify the mpi runner and parameters through the RUNNER env variable:
https://github.com/soedinglab/mmseqs2#how-to-run-mmseqs2-on-multiple-servers-using-mpi

This should work, is however not very well tested anymore since we now exclusively use high-core count servers ourselves and don't require MPI anymore.

[jamdodot]

Thank you for your reply. Can I directly modify the number of threads used by metaeuk? For example, I have 128 threads and only want to use 8, which is convenient for comparison with x86

[milot-mirdita]

Yes you can specify the --threads parameter.

[jamdodot]

I modified the /tests/test.sh file, but it seems that some commands support this parameter, some are still 128 threads.

some 128 threads

some 8 threads

test.sh

[milot-mirdita]

That sounds like a bug from our side. As a workaround, you can also set the MMSEQS_NUM_THREADS environment variable. That should globally restrict the number of threads:

MMSEQS_NUM_THREADS=8 ./tests/test.sh

[jamdodot]

This works， thanks！

[jamdodot]

This is not a problem, this is expected behavior. To benchmark MetaEuk's runtime you need to run it on larger query/target sets.

Do you have larger query/target sets. I don't know where to find a larger test set and the corresponding commands