Update OpenFOAM demo #32

laraPPr · 2024-04-19T10:45:20Z

No description provided.

laraPPr · 2024-04-19T11:28:46Z

The OpenFOAM demo can trigger this issue with v2023.06 of the EESSI production repo.

Also opened a pr to the docs with the workaround: EESSI/docs#172

satishskamath · 2024-04-25T09:22:10Z

OpenFOAM/bike_OpenFOAM8.sh

+surfaceFeatures 2>&1 | tee log.surfaceFeatures
+blockMesh 2>&1 | tee log.blockMesh
+decomposePar -copyZero 2>&1 | tee log.decomposePar
+mpirun -np $NP -ppn $PPN -hostfile hostlist snappyHexMesh -parallel -overwrite 2>&1 | tee log.snappyHexMesh


A hostfile is a must here without which the mpirun command would fail.

Why is this true here but not below?

Rather than try to support multi-node in the demos, I think it would be better to just add the disclaimer that they are prepared for single node use cases, but could be extended to multi-node cases. If you submit this job via SLURM you wouldn't need the complication of a hostfile

As this stands, I don't think it would work would it since there is no hostlist file?

OpenFOAM/bike_OpenFOAM_11.sh

…perate pr.

ocaisa

Output file should be removed

ocaisa · 2024-04-30T15:20:19Z

OpenFOAM/bike_OpenFOAM8.sh

+surfaceFeatures 2>&1 | tee log.surfaceFeatures
+blockMesh 2>&1 | tee log.blockMesh
+decomposePar -copyZero 2>&1 | tee log.decomposePar
+mpirun -np $NP -ppn $PPN -hostfile hostlist snappyHexMesh -parallel -overwrite 2>&1 | tee log.snappyHexMesh


Suggested change

mpirun -np $NP -ppn $PPN -hostfile hostlist snappyHexMesh -parallel -overwrite 2>&1 | tee log.snappyHexMesh

mpirun -np $NP --oversubscribe snappyHexMesh -parallel -overwrite 2>&1 | tee log.snappyHexMesh

Yes, I have removed this in my latest script which @laraPPr is still testing. Her next commit should make things clear.

In Satish script for version 11 it is changed to this mpirun -np $NP -npernode $PPN snappyHexMesh -parallel -overwrite 2>&1 | tee log.snappyHexMesh. And that one now works correctly. I'm also less inclined to change this one because it still does something and is only their for the OpenFOAM in the pilot repo.

I don't really have a problem with this (since it will soon be lost to history anyway), but the file hostlist does not exist as far as I can see, so it really does nothing (or is it just a workaround for the use of -ppn?).

Actually -ppn is deprecated and -npernode is to be used instead with mpirun. Indeed the hostlist thing didn't make any sense to me so I also removed it as @ocaisa did. We can also get rid of --oversubscribe option and just declare that one needs to have at least 8 cores ( I would say we even need more) to run this script.

ocaisa · 2024-05-01T15:30:57Z

Can we change OpenFOAM8.sh to OpenFOAM_v8.sh and OpenFOAM_11.sh to OpenFOAM_v11.sh?

boegel · 2024-05-01T17:14:28Z

There's a lot going on in this PR, and there may be some parts in the current demo script that isn't fully correct.

Rather than trying to fix all problems in one go, we should try and get the minimal things done so the OpenFOAM demo can be run using the OpenFOAM installations available in software.eessi.io.

This demo is by no means meant to be a perfect example of running OpenFOAM, so in the short term it's fine if some of the substeps aren't currently doing what they should be doing. The main thing is that we can show people that the OpenFOAM installation provided by EESSI runs (not that we're experts in OpenFOAM).

So, perhaps we should open a separate PR first with the minimal work to do to get the OpenFOAM demo running with software.eessi.io, and open an issue with an overview of the problems to fix?

ocaisa

Tested with software.eessi.io and worked just fine

OpenFOAM/bike_OpenFOAM_v11.sh

ocaisa · 2024-05-02T08:24:53Z

As a side point, on 4 cores I actually got an MPI error from snappyHexMesh, which made me notice that we don't actually check the return codes before proceeding to the next step.

Having said that, I agree with @boegel that we should just get this merged and worry about tweaking the scripts later (or never 😛 ).

satishskamath · 2024-05-02T09:16:46Z

As a side point, on 4 cores I actually got an MPI error from snappyHexMesh, which made me notice that we don't actually check the return codes before proceeding to the next step.

Having said that, I agree with @boegel that we should just get this merged and worry about tweaking the scripts later (or never 😛 ).

Oh yeah, we do not check the error codes from the previous steps at all. :D Btw what was the error. I am curious. :)

ocaisa · 2024-05-02T09:28:48Z

Oh yeah, we do not check the error codes from the previous steps at all. :D Btw what was the error. I am curious. :)

Surface refinement iteration 7
------------------------------

Marked for refinement due to surface intersection          : 3543 cells.
Marked for refinement due to curvature/regions             : 1282668 cells.
Determined cells to refine in = 1.43973 s
Selected for refinement : 1458107 cells (out of 7448582)
Balanced mesh in = 16.9464 s
After balancing surface refinement iteration 7 : cells:7448582  faces:24702264  points:9838521
Cells per refinement level:
    0   159607
    1   1429
    2   5209
    3   27745
    4   163998
    5   777950
    6   3596644
    7   2716000
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 3 with PID 275331 on node node1 exited on signal 9 (Killed).
--------------------------------------------------------------------------

laraPPr · 2024-05-02T10:21:22Z

Ok So I now remove the --oversubscribe flag everywhere and I added it to the run.sh script. And also remove the -hosfile flag form the bike_OpenFOAM_v8.sh script.

OpenFOAM/run.sh

Co-authored-by: ocaisa <alan.ocais@cecam.org>

laraPPr · 2024-05-03T07:44:24Z

@ocaisa is it now ok? Or should I change something else?

boegel · 2024-05-03T08:44:44Z

Oh yeah, we do not check the error codes from the previous steps at all. :D Btw what was the error. I am curious. :)

Surface refinement iteration 7
------------------------------
...
--------------------------------------------------------------------------
mpirun noticed that process rank 3 with PID 275331 on node node1 exited on signal 9 (Killed).
--------------------------------------------------------------------------

That looks like a segfault due to memory limit?

ocaisa · 2024-05-03T08:45:59Z

Oh yeah, we do not check the error codes from the previous steps at all. :D Btw what was the error. I am curious. :)
Surface refinement iteration 7
------------------------------
...
--------------------------------------------------------------------------
mpirun noticed that process rank 3 with PID 275331 on node node1 exited on signal 9 (Killed).
--------------------------------------------------------------------------
That looks like a segfault due to memory limit?

Could be, I didn't investigate

OpenFOAM/bike_OpenFOAM_v11.sh

OpenFOAM/run.sh

laraPPr · 2024-05-06T08:43:15Z

@boegel I updated the PR

ocaisa · 2024-05-08T20:08:52Z

I'm merging this and following up with a PR to drop the use of the pilot

vsc46128 vscuser and others added 2 commits April 2, 2024 13:31

load the right module for each repo

1ebd3be

update OpenFOAM demo's

24359b1

satishskamath reviewed Apr 25, 2024

View reviewed changes

OpenFOAM/bike_OpenFOAM_11.sh Outdated Show resolved Hide resolved

vsc46128 vscuser and others added 3 commits April 30, 2024 16:25

Update OpenFOAM demo

448cfd4

update motorbike script

300f142

Am also working on ESPresso demo so accidently added it. Will open se…

8ca4ba3

…perate pr.

ocaisa requested changes Apr 30, 2024

View reviewed changes

ocaisa reviewed Apr 30, 2024

View reviewed changes

vsc46128 vscuser added 2 commits May 2, 2024 09:21

remove output and rename scripts

67d9493

change names of scripts

e3d71e2

ocaisa approved these changes May 2, 2024

View reviewed changes

ocaisa requested changes May 2, 2024

View reviewed changes

OpenFOAM/bike_OpenFOAM_v11.sh Show resolved Hide resolved

set oversubscribe environment variable and remove hostfile flag from v8

f1587a9

ocaisa requested changes May 2, 2024

View reviewed changes

OpenFOAM/run.sh Outdated Show resolved Hide resolved

Update OpenFOAM/run.sh

4692bb5

Co-authored-by: ocaisa <alan.ocais@cecam.org>

boegel reviewed May 3, 2024

View reviewed changes

OpenFOAM/bike_OpenFOAM_v11.sh Show resolved Hide resolved

boegel requested changes May 3, 2024

View reviewed changes

OpenFOAM/run.sh Outdated Show resolved Hide resolved

vsc46128 vscuser added 2 commits May 6, 2024 07:32

update OpenFORM run.sh script

7a4e21c

merge with upstream branch

4741c27

ocaisa merged commit 684ef83 into EESSI:main May 8, 2024
8 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update OpenFOAM demo #32

Update OpenFOAM demo #32

laraPPr commented Apr 19, 2024

laraPPr commented Apr 19, 2024

satishskamath Apr 25, 2024

ocaisa Apr 30, 2024

ocaisa Apr 30, 2024

ocaisa Apr 30, 2024

ocaisa left a comment

ocaisa Apr 30, 2024

satishskamath May 1, 2024

laraPPr May 2, 2024

ocaisa May 2, 2024

satishskamath May 2, 2024 •

edited

ocaisa commented May 1, 2024

boegel commented May 1, 2024

ocaisa left a comment

ocaisa commented May 2, 2024

satishskamath commented May 2, 2024

ocaisa commented May 2, 2024

laraPPr commented May 2, 2024

laraPPr commented May 3, 2024

boegel commented May 3, 2024

ocaisa commented May 3, 2024

laraPPr commented May 6, 2024

ocaisa commented May 8, 2024

	mpirun -np $NP -ppn $PPN -hostfile hostlist snappyHexMesh -parallel -overwrite 2>&1 \| tee log.snappyHexMesh
	mpirun -np $NP --oversubscribe snappyHexMesh -parallel -overwrite 2>&1 \| tee log.snappyHexMesh

Update OpenFOAM demo #32

Update OpenFOAM demo #32

Conversation

laraPPr commented Apr 19, 2024

laraPPr commented Apr 19, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ocaisa left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

satishskamath May 2, 2024 • edited

Choose a reason for hiding this comment

ocaisa commented May 1, 2024

boegel commented May 1, 2024

ocaisa left a comment

Choose a reason for hiding this comment

ocaisa commented May 2, 2024

satishskamath commented May 2, 2024

ocaisa commented May 2, 2024

laraPPr commented May 2, 2024

laraPPr commented May 3, 2024

boegel commented May 3, 2024

ocaisa commented May 3, 2024

laraPPr commented May 6, 2024

ocaisa commented May 8, 2024

satishskamath May 2, 2024 •

edited