Multi-Threading not working

[Hyp5891]

Hi all,

I am using the gui job submission tool for windows from here and LS Prepost (.k-file) for preprocessing.

Multithreading (for example "-nt parameter" set to 4) works/worked without a problem for some models. I am not totally sure yet but I think after adding NRBs (rigid body connections) I cannot run anything else than one thread only (which is obiously slower). When I put higher numbers the solver is not running. Either it stops after the first timestep with no error message or with Message ID44: "Segmentation Violation".

Does anybody else had this issue and knows how to resolve this?

Thanks a lot in advance!

Best regards,
Christian

[MarianBulla]

Hi Hyp5891,
thanks for your interest in OpenRadioss :-)
It's hard to figure out the issue.
Can you add your model, so it can be deeper investigated, please ?

If it is confidential or too large, can you create a smaller one, where this issue also occures ?

Thanks and best regards,
Marian

[ktommy2]

Hi @OlivierWienholtz, hi @MarianBulla!
Thank you for your asap support.

I know that the uploaded model is small. Originally I had a droptest model with over 250k elements, but for testing/debugging purpose I created a smaller one that reproduces the same problem.

I tried to edit the KMP_Affinity. To start a simulation, i use the executable called openradioss_run_script_for_windows_mpi_ps.
Therefore i modified the batch file (openradioss_run_script_ps.bat) and pasted Olivers code.
I then did a little experiment with different policies. I did one run with each policy (and I closed all my applications during the simulations...)
Here are the results:

Ground zero: 1 thread, ET = 88.3s
KMP_AFFINITY=disabled or none, 6 threads, ET = 365.15s
KMP_AFFINITY=disabled or none, 4 threads, ET = 283.72s
KMP_AFFINITY=disabled or none, 2 threads, ET = 180.18s
KMP_AFFINITY=compact, 6 threads, ET = 291.15s
KMP_AFFINITY=scatter, 6 threads, ET = 257.81s
KMP_AFFINITY=balanced, 6 threads, ET = 251.83s

[ktommy2]

Im definitely not an expert in this topic, but mapping them didnt help either:
set KMP_AFFINITY=explicit, proclist=[0,1,2,3,4,5]

(... i hope, ive done it all right)

[ktommy2]

Hi,

i gave another try on my private workstation.
It has an Intel(R) Core(TM) i7-8700K (@ 3.70GHz (x86_64), 3700 MHz, 55301 MB RAM, 75172 MB swap).
I did not set the kmp_affinity for now.

1 thread, ET 131.30 sec
2 threads, ET 94.35 sec
6 threads, ET 83.64 sec
12 threads, ET 106.03 sec (For this small model, this setting was rather counterproductive)

I can therefore confirm that I have the problem on my company notebook and not on my workstation. Is there still a workaround? I will apply for a driver update and report if there is an improvement... damn.

[ktommy2]

Hi @MarianBulla, hi @OlivierWienholtz!

a few days have passed and i wanted to ask if you have any other ideas i could try. The update did not bring any speedup.

[OlivierWienholtz]

Hello Tommy,
We are currently investigating your issue, in particular trying to run on similar Machine than you.
We keep you in touch.

Best Regards,
Olivier

[MarianBulla]

Hi Ktommy2,
yes, I agree with Olivier,
the performance is mostly on the used hardware and the model size.
Your model is quite small.
So using more threads or cores does not make much sense/difference.
Anyway, the performance should not drop.
I have an older CPU, without the efficency cores, and I did the benchmark, using your model.
Here ist the result on my machine:

My CPU: https://www.intel.com/content/www/us/en/products/sku/213799/intel-core-i711850h-processor-24m-cache-up-to-4-80-ghz/specifications.html

Your CPU: https://www.intel.com/content/www/us/en/products/sku/232144/intel-core-i913900hk-processor-24m-cache-up-to-5-40-ghz/specifications.html

Hope the change of the KMP_AFFINITY will help you.
Please let us know, how much will bring.

Thanks and best regards,
Marian

[Hyp5891]

Hey,

I am thinking about buying a Lenovo P16 Gen2 workstation laptop. Now this recent comments on CPU performance got my attention.
Can anyone rule out that the Intel® Core™ i7-13700HX or Core™ i7-13850HX can become a problem with OpenRadioss??

Best regards,
Christian

[MarianBulla]

Hi Christian,
there is no problem with intel i7 -13700HX CPUs and OpenRadioss.
It is only a question of performance.

The newer laptop CPUs are more power efficient.

But the older one still has the AVX512, which provides ~20% more performance compared to AVX2 (see the data sheet ):

So, if you can get the "older" intel i7 11850H laptop with sufficient memory, it will also do the job very well.

Anyway, both have only 2 memory channels, which will limit the performance.

For more details with the recent intel CPUs, see the response from @OlivierWienholtz, on how to get the best out of it.
The other suggestion would be to go with a workstation instead of a laptop.

Hope it helps you ?

Thansk and best regards,
Marian

[OlivierWienholtz]

Hello

@ktommy2

We have done tests on a similar machine than your, it is a 12th Generation I7 with 6 performance (including HyperThreading)+ 8 Efficient cores.
It is similar machine than you only 1 Generation older.
I reproduced your behave regarding the SMP Performance. 2 or more threads are quite slower than 1 on your Model.
I observe same ratio than you.

Interestingly, an MPI run with 2 MPI processes give 1.3 Speedup,
4 MPI processes give 1.15 from 1 to 4, slightly worse than 2 MPI.

I tried a bigger model with 1 Million finite Elements.
https://openradioss.atlassian.net/wiki/spaces/OPENRADIOSS/pages/47546369/HPC+Benchmark+Models#1M-Element-Neon-model
I have expected speedup for such architectures:
1.8x from 1 to 2
1.6x from 2 to 4

12th & 13th Generation Laptop CPUs shows this behave on your small model,
The older 11th Generation Laptop CPUs (Rocket Lake) gives expected SMP performance.

Process Placement on such architecture has also importance: Indeed using Efficient cores slows down the job...

To know : You can guess what are the performance & Efficient cores using the cpuinfo tool shipped with IntelMPI (you have it when installing OneAPI compiler + IntelMPI):
Performance cores have Threads attached, not the Efficient cores. The "processors" with thread ID=0 from 0-11 are the 6 Performance cores.

You can use SMP Pinning to stick your commutation to a core :
I used OMP_PLACES variables, They are equivalent to KMP_ Variables.
If you set
OMP_PLACES=0,2

This will stick your SMP threads on logical cpuid=0 & 2 which are 2 P-cores
You will observe performance slowdown if you run in Mono on core 12 to 19.

Hope it helps,
Olivier W.

[OlivierWienholtz]

We continued investigating :
You may try to set following environment variable on your test system :

set OMP_WAIT_POLICY=ACTIVE

On the machine I am testing, I could achieve
1 Thread : 90 s
2 Threads : 67s
4 Threads : 60s
6 Threads : 63s
The model remains small it is to expect that I won't scale but there is no degradation.

[ktommy2]

Hello, sorry for the late reply. I was busy and got sick last week....
The suggested modification to the batch file seems to work well. Thank you very much for that!
Next week I will also test it with a larger model and give you a feedback. After that, we can close the topic and I accept your answer as a solution to this problem!
Thank you for your commitment!