Add initial Apptainer/Singularity definition file

[Irvise]

Hi ElmerFEM team.

I created an Apptainer/Singularity definition file for Elmer for me to test it and have a "stable/reliable" way of building it. Here is the version I currently have. I build it with everything and I try as much as possible to link with the system libraries (this may not be desirable in some scenarios). This means it is MPI ready and it should have access to all the solvers/meshers. The image is currently quite raw. For example I do not clean the build directory (because I want to be able to test ElmerFEM) and I install a few things that may not be necessary. For example I install the system UMFPACK, but ElmerFEM builds its own version.

You can try and build it with apptainer build elmer.sif elmer.def. However, I would actually recommend to build it in sandbox mode in order to test it and debug it. This can be done by building with apptainer build --sandbox elmer_testing elmer.def. This will create a folder where the entire image resides. We can run ElmerFEM by running apptainer run -B /run elmer_testing and ElmerGUI should show up. I had to add the option -B /run in order for the GUI to detect/attach to the running display.

In order to run the tests, use apptainer shell --writable elmer_testing. This will drop you into a shell within the container/folder and will make it writable (which is need in order to run the tests). Then, inside the shell session just do a cd /root/build && make test.

Currently, it shows me that there are 1040 available tests. If I remember correctly, in the past it told me it could run more, but maybe I am missremembering or things may have changed. THERE ARE QUITE A FEW FAILING TESTS... I am not good enough with Elmer to debug them... I do remember that after running all of them and then rerunning the failing ones, some did pass. I do not know if this is an issues with running make test -j14 or if there are memory bugs that only show up sporadically or something else... Maybe it has to do with me using some 64bit int math libraries...

Anyhow, here is the initial draft. I would like to receive feedback and criticism. I am specially interested in why there are quite a few failing tests... I also hope that this helps the community and testing of ElmerFEM. And of course, you can propose changes, such as changing Debian for another base system or whatever :)

Best regards,
Fer

[Irvise]

Pinging @ettaka

[Irvise]

Here are some failing tests as I am running now the suite, which does take a while in my computer...

fernando@localhost:~/Build/elmerfem/docker> apptainer shell --writable -B /run elmer_sandbox/
Apptainer> cd /root/build/
Apptainer> make test -j14
Running tests...
Test project /root/build
          Start    1: AIFlowSolve
   1/1040 Test    #1: AIFlowSolve .............................................***Failed    0.54 sec
          Start    2: Buoyancy
   2/1040 Test    #2: Buoyancy ................................................   Passed    0.49 sec
          Start    3: Calving2D
   3/1040 Test    #3: Calving2D ...............................................   Passed    7.09 sec
          Start    4: Calving3D
   4/1040 Test    #4: Calving3D ...............................................***Failed    7.67 sec
          Start    5: ComputeDevStress
   5/1040 Test    #5: ComputeDevStress ........................................   Passed    0.66 sec
          Start    6: Contact
   6/1040 Test    #6: Contact .................................................   Passed    4.55 sec
          Start    7: DGsolver
   7/1040 Test    #7: DGsolver ................................................   Passed    1.38 sec
          Start    8: Damage
   8/1040 Test    #8: Damage ..................................................***Failed   43.14 sec
          Start    9: Dating
   9/1040 Test    #9: Dating ..................................................   Passed    1.12 sec
          Start   10: Density
  10/1040 Test   #10: Density .................................................   Passed    4.16 sec
          Start   11: EigenValues
  11/1040 Test   #11: EigenValues .............................................   Passed    0.94 sec
          Start   12: Emergence
  12/1040 Test   #12: Emergence ...............................................   Passed    0.86 sec

[...]

  35/1040 Test   #35: Grid2DInterpolator ......................................   Passed    0.99 sec
          Start   36: Grounded
  36/1040 Test   #36: Grounded ................................................   Passed    1.25 sec
          Start   37: Hydro_Coupled
  37/1040 Test   #37: Hydro_Coupled ...........................................***Failed   24.58 sec
          Start   38: Hydro_SedOnly
  38/1040 Test   #38: Hydro_SedOnly ...........................................***Failed   28.89 sec
          Start   39: IntegrateVertically
  39/1040 Test   #39: IntegrateVertically .....................................   Passed    0.76 sec
          Start   40: IntegratedVelocity
  40/1040 Test   #40: IntegratedVelocity ......................................   Passed    0.42 sec
          Start   41: InvMeth_AdjRobin
  41/1040 Test   #41: InvMeth_AdjRobin ........................................***Failed   38.54 sec
          Start   42: LateralFriction
  42/1040 Test   #42: LateralFriction .........................................   Passed    0.81 sec
          Start   43: MISMIP_FS-SSA
  43/1040 Test   #43: MISMIP_FS-SSA ...........................................***Failed   49.11 sec
          Start   44: MMG2D_Aniso1
  44/1040 Test   #44: MMG2D_Aniso1 ............................................   Passed    0.48 sec
          Start   45: MMG2D_Aniso2
  45/1040 Test   #45: MMG2D_Aniso2 ............................................   Passed    0.64 sec
          Start   46: MMG2D_Iso
  46/1040 Test   #46: MMG2D_Iso ...............................................   Passed    0.46 sec
          Start   47: MMG2D_Transient
  47/1040 Test   #47: MMG2D_Transient .........................................***Failed    1.40 sec
          Start   48: Permafrost_Biot
  48/1040 Test   #48: Permafrost_Biot .........................................***Failed    5.36 sec
          Start   49: Permafrost_Frozenwall
  49/1040 Test   #49: Permafrost_Frozenwall ...................................   Passed   13.40 sec

[...]

         Start  347: Hybrid2dMeshPartitionCyl_np8
 347/1040 Test  #347: Hybrid2dMeshPartitionCyl_np8 ............................   Passed    1.06 sec
          Start  348: Hybrid2dMeshPartitionMetis_np8
 348/1040 Test  #348: Hybrid2dMeshPartitionMetis_np8 ..........................   Passed    1.02 sec
          Start  349: Hybrid2dMeshPartitionMetisConnect_np8
 349/1040 Test  #349: Hybrid2dMeshPartitionMetisConnect_np8 ...................   Passed    1.09 sec
          Start  350: HybridMeshMap
 350/1040 Test  #350: HybridMeshMap ...........................................   Passed    2.67 sec
          Start  351: HydrostaticNSVec-ISMIP-HOM-C
 351/1040 Test  #351: HydrostaticNSVec-ISMIP-HOM-C ............................   Passed   21.08 sec
          Start  352: IncreaseOrderBrick
 352/1040 Test  #352: IncreaseOrderBrick ......................................   Passed    2.46 sec
          Start  353: IncreaseOrderPrism
 353/1040 Test  #353: IncreaseOrderPrism ......................................   Passed    2.44 sec
          Start  354: IncreaseOrderQuad
 354/1040 Test  #354: IncreaseOrderQuad .......................................   Passed    2.43 sec
          Start  355: IncreaseOrderTet
 355/1040 Test  #355: IncreaseOrderTet ........................................   Passed    2.45 sec
          Start  356: IncreaseOrderTri
 356/1040 Test  #356: IncreaseOrderTri ........................................   Passed    2.42 sec
          Start  357: InductionHeating
 357/1040 Test  #357: InductionHeating ........................................   Passed    2.52 sec
          Start  358: InductionHeating2
 358/1040 Test  #358: InductionHeating2 .......................................   Passed    2.77 sec
          Start  359: InductionHeating3
 359/1040 Test  #359: InductionHeating3 .......................................   Passed    2.55 sec
          Start  360: IntegralConstraintFlow
 360/1040 Test  #360: IntegralConstraintFlow ..................................   Passed    2.45 sec
          Start  361: IntegralConstraintHartmann
 361/1040 Test  #361: IntegralConstraintHartmann ..............................   Passed    4.58 sec
          Start  362: IntegralConstraintPoisson
 362/1040 Test  #362: IntegralConstraintPoisson ...............................   Passed    2.43 sec
          Start  363: InternalPartitioning
 363/1040 Test  #363: InternalPartitioning ....................................   Passed    3.23 sec
          Start  364: InternalPartitioning_np8
 364/1040 Test  #364: InternalPartitioning_np8 ................................***Failed    1.86 sec
          Start  365: InternalPartitioning2
 365/1040 Test  #365: InternalPartitioning2 ...................................   Passed    3.30 sec
          Start  366: InternalPartitioning2_np6
 366/1040 Test  #366: InternalPartitioning2_np6 ...............................***Failed    1.75 sec
          Start  367: InternalPartitioning3
 367/1040 Test  #367: InternalPartitioning3 ...................................   Passed    3.25 sec
          Start  368: InternalPartitioning3_np6
 368/1040 Test  #368: InternalPartitioning3_np6 ...............................***Failed    1.70 sec
          Start  369: InternalPartitioning4
 369/1040 Test  #369: InternalPartitioning4 ...................................   Passed    2.64 sec
          Start  370: InternalPartitioning5
 370/1040 Test  #370: InternalPartitioning5 ...................................   Passed    2.62 sec
          Start  371: InternalPartitioning6
 371/1040 Test  #371: InternalPartitioning6 ...................................   Passed    2.58 sec

[...]

And I just want to point out that the failing tests are not failing due to my computer not having enough CPUs ;)

[Irvise]

HYPRE seems to be broken in this build.

          Start  744: WinkelBmPoissonCgIlu0_np2
 744/1040 Test  #744: WinkelBmPoissonCgIlu0_np2 ...............................   Passed    0.81 sec
          Start  745: WinkelBmPoissonCgIlu0_np8
 745/1040 Test  #745: WinkelBmPoissonCgIlu0_np8 ...............................   Passed    0.70 sec
          Start  746: WinkelBmPoissonHypreBiCGStabBoomer_np2
 746/1040 Test  #746: WinkelBmPoissonHypreBiCGStabBoomer_np2 ..................***Failed    6.03 sec
          Start  747: WinkelBmPoissonHypreBiCGStabBoomer_np8
 747/1040 Test  #747: WinkelBmPoissonHypreBiCGStabBoomer_np8 ..................***Failed    0.61 sec
          Start  748: WinkelBmPoissonHypreBiCGStabIlu0_np2
 748/1040 Test  #748: WinkelBmPoissonHypreBiCGStabIlu0_np2 ....................***Failed    0.58 sec
          Start  749: WinkelBmPoissonHypreBiCGStabIlu0_np8
 749/1040 Test  #749: WinkelBmPoissonHypreBiCGStabIlu0_np8 ....................***Failed    0.59 sec
          Start  750: WinkelBmPoissonHypreBiCGStabIlu1_np2
 750/1040 Test  #750: WinkelBmPoissonHypreBiCGStabIlu1_np2 ....................***Failed    0.58 sec
          Start  751: WinkelBmPoissonHypreBiCGStabIlu1_np8
 751/1040 Test  #751: WinkelBmPoissonHypreBiCGStabIlu1_np8 ....................***Failed    0.64 sec
          Start  752: WinkelBmPoissonHypreBoomer_np2
 752/1040 Test  #752: WinkelBmPoissonHypreBoomer_np2 ..........................***Failed    1.58 sec
          Start  753: WinkelBmPoissonHypreBoomer_np8
 753/1040 Test  #753: WinkelBmPoissonHypreBoomer_np8 ..........................***Failed    0.69 sec
          Start  754: WinkelBmPoissonHypreCgIlu0_np2
 754/1040 Test  #754: WinkelBmPoissonHypreCgIlu0_np2 ..........................***Failed    0.58 sec
          Start  755: WinkelBmPoissonHypreCgIlu0_np8
 755/1040 Test  #755: WinkelBmPoissonHypreCgIlu0_np8 ..........................***Failed    0.61 sec
          Start  756: WinkelBmPoissonHypreFlexGMResILU0_np2
 756/1040 Test  #756: WinkelBmPoissonHypreFlexGMResILU0_np2 ...................***Failed    1.58 sec
          Start  757: WinkelBmPoissonHypreFlexGMResILU0_np8
 757/1040 Test  #757: WinkelBmPoissonHypreFlexGMResILU0_np8 ...................***Failed    0.71 sec
          Start  758: WinkelBmPoissonHypreGMResBoomer_np2
 758/1040 Test  #758: WinkelBmPoissonHypreGMResBoomer_np2 .....................***Failed    0.58 sec
          Start  759: WinkelBmPoissonHypreGMResBoomer_np8
 759/1040 Test  #759: WinkelBmPoissonHypreGMResBoomer_np8 .....................***Failed    0.63 sec
          Start  760: WinkelBmPoissonHypreGMResILU0_np2
 760/1040 Test  #760: WinkelBmPoissonHypreGMResILU0_np2 .......................***Failed    4.91 sec
          Start  761: WinkelBmPoissonHypreGMResILU0_np8
 761/1040 Test  #761: WinkelBmPoissonHypreGMResILU0_np8 .......................***Failed    0.62 sec
          Start  762: WinkelBmPoissonIdrsIlu0_np2
 762/1040 Test  #762: WinkelBmPoissonIdrsIlu0_np2 .............................   Passed    0.68 sec
          Start  763: WinkelBmPoissonIdrsIlu0_np8
 763/1040 Test  #763: WinkelBmPoissonIdrsIlu0_np8 .............................   Passed    0.69 sec

[Irvise]

Here is the summary of the results

96% tests passed, 42 tests failed out of 1040

Label Time Summary:
2D                =  59.61 sec*proc (19 tests)
3D                =  62.97 sec*proc (15 tests)
aster             =  20.46 sec*proc (7 tests)
battery           =   3.82 sec*proc (1 test)
beam              =   9.77 sec*proc (4 tests)
benchmark         = 116.49 sec*proc (18 tests)
block             = 123.45 sec*proc (37 tests)
circuit_utils     =   2.49 sec*proc (1 test)
circuits          =  95.11 sec*proc (25 tests)
cmodes            = 150.72 sec*proc (24 tests)
conforming        =  24.17 sec*proc (1 test)
contact           =  94.15 sec*proc (24 tests)
control           =  46.75 sec*proc (18 tests)
dg                =   3.54 sec*proc (3 tests)
eigen             =   9.11 sec*proc (3 tests)
elasticity        =  50.11 sec*proc (10 tests)
elasticsolve      = 227.14 sec*proc (45 tests)
eliminate         =   4.17 sec*proc (4 tests)
elmerice          = 685.98 sec*proc (65 tests)
elmerice-fast     =  95.01 sec*proc (36 tests)
elmerice-long     = 355.82 sec*proc (8 tests)
em-wave           =  17.18 sec*proc (5 tests)
extrude           =  21.92 sec*proc (9 tests)
fsi               =  54.31 sec*proc (16 tests)
gauge             =  29.32 sec*proc (3 tests)
harmonic          =  89.55 sec*proc (27 tests)
heateq            = 110.43 sec*proc (47 tests)
helmholtz         =  23.98 sec*proc (9 tests)
homogenization    =   7.33 sec*proc (2 tests)
hutiter           =   2.87 sec*proc (4 tests)
hypre             =  22.58 sec*proc (18 tests)
lua               =  36.04 sec*proc (12 tests)
lumping           =  51.77 sec*proc (20 tests)
matc              = 105.03 sec*proc (35 tests)
mgdyn             = 185.22 sec*proc (44 tests)
mmg               = 134.16 sec*proc (14 tests)
mortar            = 280.04 sec*proc (71 tests)
mumps             =  19.82 sec*proc (9 tests)
n-t               = 162.17 sec*proc (35 tests)
namespace         =  20.68 sec*proc (8 tests)
p-fem             =  85.47 sec*proc (15 tests)
parallel          = 203.55 sec*proc (130 tests)
particle          =  54.98 sec*proc (17 tests)
permafrost        =  18.76 sec*proc (2 tests)
plate             =  30.16 sec*proc (11 tests)
quick             = 1032.95 sec*proc (409 tests)
radiation         =   4.11 sec*proc (1 test)
radiator          =  50.71 sec*proc (8 tests)
resistor          =   2.56 sec*proc (1 test)
restart           =  57.31 sec*proc (21 tests)
rotm              =   6.32 sec*proc (1 test)
scanning          =   8.45 sec*proc (2 tests)
serendipity       = 166.93 sec*proc (36 tests)
serial            = 3295.03 sec*proc (846 tests)
shell             = 216.16 sec*proc (39 tests)
slow              = 801.72 sec*proc (34 tests)
stranded          =   8.64 sec*proc (2 tests)
thin_region       =   8.59 sec*proc (3 tests)
threaded          =  55.62 sec*proc (9 tests)
transient         = 205.88 sec*proc (57 tests)
umat              =   7.54 sec*proc (3 tests)
useextrude        =  57.66 sec*proc (24 tests)
vector_element    =  39.24 sec*proc (14 tests)
viscoelastic      =  25.18 sec*proc (2 tests)
vtu               =  60.46 sec*proc (13 tests)
whitney           = 415.60 sec*proc (79 tests)
wvector           =   6.06 sec*proc (1 test)

Total Test time (real) = 4060.10 sec

The following tests FAILED:
          1 - AIFlowSolve (Failed)
          4 - Calving3D (Failed)
          8 - Damage (Failed)
         20 - FrictionHeatMasked (Failed)
         37 - Hydro_Coupled (Failed)
         38 - Hydro_SedOnly (Failed)
         41 - InvMeth_AdjRobin (Failed)
         43 - MISMIP_FS-SSA (Failed)
         47 - MMG2D_Transient (Failed)
         48 - Permafrost_Biot (Failed)
        194 - CurvedBoundaryCylH_np3 (Failed)
        195 - CurvedBoundaryCylH_np8 (Failed)
        257 - ElastPelem2dPmultgBoomer_np3 (Failed)
        258 - ElastPelem2dPmultgMumps_np3 (Failed)
        364 - InternalPartitioning_np8 (Failed)
        366 - InternalPartitioning2_np6 (Failed)
        368 - InternalPartitioning3_np6 (Failed)
        561 - RotatingBlockRemeshMMG3D (Failed)
        657 - ShoeboxFsiStatic (Failed)
        659 - ShoeboxFsiStaticShell (Failed)
        677 - StatElecSolveMMG3D (Failed)
        679 - StatElecSolveVecMMG3D (Failed)
        740 - WinkelBmNavierHypreBiCGStabBoomer_np4 (Failed)
        746 - WinkelBmPoissonHypreBiCGStabBoomer_np2 (Failed)
        747 - WinkelBmPoissonHypreBiCGStabBoomer_np8 (Failed)
        748 - WinkelBmPoissonHypreBiCGStabIlu0_np2 (Failed)
        749 - WinkelBmPoissonHypreBiCGStabIlu0_np8 (Failed)
        750 - WinkelBmPoissonHypreBiCGStabIlu1_np2 (Failed)
        751 - WinkelBmPoissonHypreBiCGStabIlu1_np8 (Failed)
        752 - WinkelBmPoissonHypreBoomer_np2 (Failed)
        753 - WinkelBmPoissonHypreBoomer_np8 (Failed)
        754 - WinkelBmPoissonHypreCgIlu0_np2 (Failed)
        755 - WinkelBmPoissonHypreCgIlu0_np8 (Failed)
        756 - WinkelBmPoissonHypreFlexGMResILU0_np2 (Failed)
        757 - WinkelBmPoissonHypreFlexGMResILU0_np8 (Failed)
        758 - WinkelBmPoissonHypreGMResBoomer_np2 (Failed)
        759 - WinkelBmPoissonHypreGMResBoomer_np8 (Failed)
        760 - WinkelBmPoissonHypreGMResILU0_np2 (Failed)
        761 - WinkelBmPoissonHypreGMResILU0_np8 (Failed)
        927 - mgdyn_hypre_ams_np2 (Failed)
        940 - mgdyn_steady_quad_extruded_restart (Failed)
        941 - mgdyn_steady_quad_extruded_restart_np3 (Failed)
Errors while running CTest
Output from these tests are in: /root/build/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
make: *** [Makefile:91: test] Error 8
Apptainer> 

Quite a few tests fail like this:

Apptainer> ctest --rerun-failed --output-on-failure
Test project /root/build
      Start   1: AIFlowSolve
 1/42 Test   #1: AIFlowSolve ..............................***Failed    0.47 sec
-- BINARY_DIR = /root/build
CMake Error at /root/elmerfem/elmerice/Tests/test_macros.cmake:76 (FILE):
  FILE failed to open for reading (No such file or directory):

    /root/build/elmerice/Tests/AIFlowSolve/TEST.PASSED
Call Stack (most recent call first):
  /root/elmerfem/elmerice/Tests/AIFlowSolve/runTest.cmake:3 (RUN_ELMERICE_TEST)

And like this, just right after running Elmergrid

      Start   4: Calving3D
 2/42 Test   #4: Calving3D ................................***Failed    5.41 sec

Starting program Elmergrid, compiled on Jan  5 2024
Elmergrid reading in-line arguments
Lower dimensional boundaries will be removed
Materials and boundaries will be renumbered
Nodes that do not appear in any element will be removed
The mesh will be partitioned with Metis to 4 partitions.
Output will be saved to file PlanMesh.

Elmergrid loading data:
-----------------------
Format chosen using the first line: $MeshFormat
Gmsh version is 2.2
Loading mesh in Gmsh format 2.0 from file PlanMesh.msh
Allocating for 282 knots and 562 elements.
Moving bulk elements to boundary elements
Leading bulk elementtype is 303
Trailing bulk elementtype is 202
There are 66 (out of 562) lower dimensional elements.
Node 77 belongs to maximum of 8 elements
Found correctly 66 side elements.
Parent elements were reordered up to index 496.
Moved 496 elements (out of 562) to new positions
Successfully read the mesh from the Gmsh input file.
Using physical numbering of entities

Elmergrid creating and manipulating meshes:
-------------------------------------------
Removing lower dimensional boundaries
Maximum elementtype is 303 and dimension 2
Minimum elementtype is 303 and dimension 2
Removed 0 (out of 66) less than 1D boundary elements
All 282 nodes were used by the mesh elements
Initial boundary interval [101,104]
BC index changed 101 -> 1 in 13 1D elements
BC index changed 102 -> 2 in 10 1D elements
BC index changed 103 -> 3 in 13 1D elements
BC index changed 104 -> 4 in 30 1D elements
Mapping boundary types from [101 104] to [1 4]
Initial body interval [107,107]
body index changed 107 -> 1 in 496 elements
Mapping material types from [107 107] to [1 1]
Renumbering of material types completed!

Elmergrid partitioning meshes:
------------------------------
Setting default Metis options!
Setting user defined Metis options!
Making a Metis partitioning for 496 elements in 3-dimensions.
All elements are of type 303
Minimum number of linear nodes in elements 3
Requiring number of nodes in dual graph 2
Using all 282 possible nodes in the Metis graph
Allocating mesh topology of size 1488
Starting graph partitioning METIS_PartMeshNodal.
 Runtime parameters:
   Objective type: METIS_OBJTYPE_CUT
   Coarsening type: METIS_CTYPE_SHEM
   Initial partitioning type: METIS_IPTYPE_METISRB
   Refinement type: METIS_RTYPE_GREEDY
   Perform a 2-hop matching: No
   Number of balancing constraints: 1
   Number of refinement iterations: 10
   Random number seed: -1
   Number of partitions: 4
   Number of cuts: 1
   User-supplied ufactor: 30
   Minimize connectivity: No
   Create contigous partitions: No
   Target partition weights: 
        0=[2.50e-01]   1=[2.50e-01]   2=[2.50e-01]   3=[2.50e-01]
   Allowed maximum load imbalance: 1.030 


 gk_mcore statistics
           coresize:         4728         nmops:         2048  cmop:      0
        num_callocs:            9   num_hallocs:            0
       size_callocs:         5840  size_hallocs:            0
        cur_callocs:            0   cur_hallocs:            0
        max_callocs:         2952   max_hallocs:            0
 nbrpool statistics
        nbrpoolsize:            0   nbrpoolcpos:            0
    nbrpoolreallocs:            0


Timing Information -------------------------------------------------
 Multilevel:               0.001
     Coarsening:                   0.000
            Matching:                      0.000
            Contract:                      0.000
     Initial Partition:            0.001
     Uncoarsening:                 0.000
          Refinement:                      0.000
          Projection:                      0.000
     Splitting:                    0.000
********************************************************************

 gk_mcore statistics
           coresize:         4728         nmops:         2048  cmop:      0
        num_callocs:           26   num_hallocs:            0
       size_callocs:         9248  size_hallocs:            0
        cur_callocs:            0   cur_hallocs:            0
        max_callocs:         4576   max_hallocs:            0
 nbrpool statistics
        nbrpoolsize:         1384   nbrpoolcpos:          440
    nbrpoolreallocs:            0

Finished graph partitioning METIS_PartMeshNodal.
Set the partition given by Metis for each node
Successfully made a Metis partition using the element mesh.
Optimizing the partitioning at boundaries.
Ownership of 0 parents was changed at BCs
Optimizing for 4 partitions
Creating a table showing all parenting partitions of nodes.
Nodes belong to 3 partitions in maximum
There are 41 shared nodes which is 14.54 % of all nodes.
The initial owner was not any of the elements for 0 nodes
Checking partitioning before optimization
Checking for partitioning
Information on partition bandwidth
Distribution of elements, nodes and shared nodes
     partition  elements   nodes      shared    
     1          128        72         14        
     2          118        70         8         
     3          123        70         11        
     4          127        70         11        
Average number of elements in partition 7.050e+01
Maximum deviation in ownership 2
Average deviation in ownership 8.660e-01
Average relative deviation 1.23 %
Checking for problematic sharings
Optimizing sharing for 4 partitions
There shouldn't be any problematic sharings, knock, knock...
Partitioning was not altered
Partitioning routines finished!

Elmergrid saving data with method 2:
-------------------------------------
Saving Elmer mesh in partitioned format
Number of boundary nodes at the boundary: 66
Number of additional interface nodes: 0
Reusing existing subdirectory: partitioning.4
Saving mesh in parallel ElmerSolver format to directory PlanMesh/partitioning.4.
Nodes belong to 3 partitions in maximum
Saving mesh for 4 partitions
   part  elements   nodes      shared   bc elems
   1     128        72         14       16      
   2     118        70         8        21      
   3     123        70         11       17      
   4     127        70         11       12      
----------------------------------------------------------------------------------------------
   ave   124.0      70.5       11.0     16.5     0.0     
Writing of partitioned mesh finished

Thank you for using Elmergrid!
Send bug reports and feature wishes to elmeradm@csc.fi
-- BINARY_DIR = /root/build
CMake Warning (dev) at /root/elmerfem/elmerice/Tests/test_macros.cmake:60 (IF):
  Policy CMP0054 is not set: Only interpret if() arguments as variables or
  keywords when unquoted.  Run "cmake --help-policy CMP0054" for policy
  details.  Use the cmake_policy command to set the policy and suppress this
  warning.

  Quoted variables like "WITH_MPI" will no longer be dereferenced when the
  policy is set to NEW.  Since the policy is not set the OLD behavior will be
  used.
Call Stack (most recent call first):
  /root/elmerfem/elmerice/Tests/Calving3D/runTest.cmake:35 (RUN_ELMERICE_TEST)
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at /root/elmerfem/elmerice/Tests/test_macros.cmake:62 (IF):
  Policy CMP0054 is not set: Only interpret if() arguments as variables or
  keywords when unquoted.  Run "cmake --help-policy CMP0054" for policy
  details.  Use the cmake_policy command to set the policy and suppress this
  warning.

  Quoted variables like "N" will no longer be dereferenced when the policy is
  set to NEW.  Since the policy is not set the OLD behavior will be used.
Call Stack (most recent call first):
  /root/elmerfem/elmerice/Tests/Calving3D/runTest.cmake:35 (RUN_ELMERICE_TEST)
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Error at /root/elmerfem/elmerice/Tests/test_macros.cmake:74 (FILE):
  FILE failed to open for reading (No such file or directory):

    /root/build/elmerice/Tests/Calving3D/TEST.PASSED_4
Call Stack (most recent call first):
  /root/elmerfem/elmerice/Tests/Calving3D/runTest.cmake:35 (RUN_ELMERICE_TEST)

[ettaka]

Great thanks! I will have a look next neek.

[tzwinger]

Latest devel branch in Elmer allows for externally provided UMFPACK - I tested with SuiteSparse. These were the configuration lines I added
-DEXTERNAL_UMFPACK:BOOL=TRUE -DUMFPACK_LIBRARIES="-L/opt/SuiteSparse/lib64/ -lumfpack -lamd -lcholmod -lsuitesparseconfig -lccolamd -lcamd -lbtf" -DUMFPACK_INCLUDE_DIR="/opt/SuiteSparse/include/suitesparse"

[Irvise]

Thanks @tzwinger! I will update the Apptainer file to reflect this change and run the tests once again :)

You may want to mention your solution on #434 I think having a toggle between prepackaged UMFPACK and external (GPL) UMFPACK is the best of both worlds.

Also, one question. Why don't we make CMake add the linker flags -lumfpack -lamd -lcholmod -lsuitesparseconfig -lccolamd -lcamd -lbtf automatically? I suppose they will always be needed if we are using the external UMFPACK. That way, the library would just behave like any other :)

Thanks,
Fer

[Irvise]

I just built the Apptainer image with your flags (adapted to use Debian's packaged libs) and it compiles. There are a few warnings regarding implicit declarations, but that is about it. Here is one of them:

/root/elmerfem/fem/src/umf4_f77wrapper.c:332:5: note: in expansion of macro 'UMFPACK_free_numeric'
  332 |     UMFPACK_free_numeric (Numeric) ;
      |     ^~~~~~~~~~~~~~~~~~~~
/root/elmerfem/fem/src/umf4_f77wrapper.c: In function 'umf4_l_fsym':
/root/elmerfem/fem/src/umf4_f77wrapper.c:238:34: warning: implicit declaration of function 'umfpack_dl_free_symbolic'; did you mean 'UMFPACK_free_symbolic'? [-Wimplicit-function-declaration]
  238 | #define UMFPACK_free_symbolic    umfpack_dl_free_symbolic
      |                                  ^~~~~~~~~~~~~~~~~~~~~~~~

I am running now the test suite once again to see if there are any new failures.

EDIT: note that I am not sure if those UMFPACK warnings were already present or if they are new. I am just pointing them out now.

[Irvise]

I can report that no new tests are failing by using the system provided UMFPACK. However, the tests took a bit longer to run eventhough I changed the make test to use 16 threads instead of 14... Here are the actual numbers:

96% tests passed, 42 tests failed out of 1040

Label Time Summary:
2D                =  64.27 sec*proc (19 tests)
3D                =  88.91 sec*proc (15 tests)
aster             =  20.56 sec*proc (7 tests)
battery           =   4.21 sec*proc (1 test)
beam              =   9.99 sec*proc (4 tests)
benchmark         = 118.67 sec*proc (18 tests)
block             = 141.00 sec*proc (37 tests)
circuit_utils     =   2.53 sec*proc (1 test)
circuits          = 124.37 sec*proc (25 tests)
cmodes            = 172.98 sec*proc (24 tests)
conforming        =  16.79 sec*proc (1 test)
contact           = 124.45 sec*proc (24 tests)
control           =  49.51 sec*proc (18 tests)
dg                =   3.80 sec*proc (3 tests)
eigen             =  10.39 sec*proc (3 tests)
elasticity        =  59.65 sec*proc (10 tests)
elasticsolve      = 276.74 sec*proc (45 tests)
eliminate         =   4.43 sec*proc (4 tests)
elmerice          = 741.41 sec*proc (65 tests)
elmerice-fast     = 149.12 sec*proc (36 tests)
elmerice-long     = 289.47 sec*proc (8 tests)
em-wave           =  17.75 sec*proc (5 tests)
extrude           =  26.52 sec*proc (9 tests)
fsi               =  59.52 sec*proc (16 tests)
gauge             =  41.62 sec*proc (3 tests)
harmonic          = 112.89 sec*proc (27 tests)
heateq            = 124.62 sec*proc (47 tests)
helmholtz         =  24.52 sec*proc (9 tests)
homogenization    =  10.34 sec*proc (2 tests)
hutiter           =   3.65 sec*proc (4 tests)
hypre             =  21.72 sec*proc (18 tests)
lua               =  35.26 sec*proc (12 tests)
lumping           =  56.26 sec*proc (20 tests)
matc              = 109.74 sec*proc (35 tests)
mgdyn             = 237.11 sec*proc (44 tests)
mmg               = 158.08 sec*proc (14 tests)
mortar            = 326.29 sec*proc (71 tests)
mumps             =  19.15 sec*proc (9 tests)
n-t               = 206.45 sec*proc (35 tests)
namespace         =  21.05 sec*proc (8 tests)
p-fem             =  70.35 sec*proc (15 tests)
parallel          = 210.08 sec*proc (130 tests)
particle          =  58.65 sec*proc (17 tests)
permafrost        =  22.45 sec*proc (2 tests)
plate             =  33.34 sec*proc (11 tests)
quick             = 1053.56 sec*proc (409 tests)
radiation         =   4.79 sec*proc (1 test)
radiator          =  60.49 sec*proc (8 tests)
resistor          =   2.57 sec*proc (1 test)
restart           =  62.60 sec*proc (21 tests)
rotm              =  12.61 sec*proc (1 test)
scanning          =   9.19 sec*proc (2 tests)
serendipity       = 209.35 sec*proc (36 tests)
serial            = 3721.15 sec*proc (846 tests)
shell             = 255.19 sec*proc (39 tests)
slow              = 796.63 sec*proc (34 tests)
stranded          =  12.65 sec*proc (2 tests)
thin_region       =   8.95 sec*proc (3 tests)
threaded          =  47.30 sec*proc (9 tests)
transient         = 229.24 sec*proc (57 tests)
umat              =   7.62 sec*proc (3 tests)
useextrude        =  57.97 sec*proc (24 tests)
vector_element    =  39.55 sec*proc (14 tests)
viscoelastic      =  32.25 sec*proc (2 tests)
vtu               =  79.17 sec*proc (13 tests)
whitney           = 552.59 sec*proc (79 tests)
wvector           =  10.32 sec*proc (1 test)

Total Test time (real) = 4489.66 sec

The following tests FAILED:
          1 - AIFlowSolve (Failed)
          4 - Calving3D (Failed)
          8 - Damage (Failed)
         20 - FrictionHeatMasked (Failed)
         37 - Hydro_Coupled (Failed)
         38 - Hydro_SedOnly (Failed)
         41 - InvMeth_AdjRobin (Failed)
         43 - MISMIP_FS-SSA (Failed)
         47 - MMG2D_Transient (Failed)
         48 - Permafrost_Biot (Failed)
        194 - CurvedBoundaryCylH_np3 (Failed)
        195 - CurvedBoundaryCylH_np8 (Failed)
        257 - ElastPelem2dPmultgBoomer_np3 (Failed)
        258 - ElastPelem2dPmultgMumps_np3 (Failed)
        364 - InternalPartitioning_np8 (Failed)
        366 - InternalPartitioning2_np6 (Failed)
        368 - InternalPartitioning3_np6 (Failed)
        561 - RotatingBlockRemeshMMG3D (Failed)
        657 - ShoeboxFsiStatic (Failed)
        659 - ShoeboxFsiStaticShell (Failed)
        677 - StatElecSolveMMG3D (Failed)
        679 - StatElecSolveVecMMG3D (Failed)
        740 - WinkelBmNavierHypreBiCGStabBoomer_np4 (Failed)
        746 - WinkelBmPoissonHypreBiCGStabBoomer_np2 (Failed)
        747 - WinkelBmPoissonHypreBiCGStabBoomer_np8 (Failed)
        748 - WinkelBmPoissonHypreBiCGStabIlu0_np2 (Failed)
        749 - WinkelBmPoissonHypreBiCGStabIlu0_np8 (Failed)
        750 - WinkelBmPoissonHypreBiCGStabIlu1_np2 (Failed)
        751 - WinkelBmPoissonHypreBiCGStabIlu1_np8 (Failed)
        752 - WinkelBmPoissonHypreBoomer_np2 (Failed)
        753 - WinkelBmPoissonHypreBoomer_np8 (Failed)
        754 - WinkelBmPoissonHypreCgIlu0_np2 (Failed)
        755 - WinkelBmPoissonHypreCgIlu0_np8 (Failed)
        756 - WinkelBmPoissonHypreFlexGMResILU0_np2 (Failed)
        757 - WinkelBmPoissonHypreFlexGMResILU0_np8 (Failed)
        758 - WinkelBmPoissonHypreGMResBoomer_np2 (Failed)
        759 - WinkelBmPoissonHypreGMResBoomer_np8 (Failed)
        760 - WinkelBmPoissonHypreGMResILU0_np2 (Failed)
        761 - WinkelBmPoissonHypreGMResILU0_np8 (Failed)
        927 - mgdyn_hypre_ams_np2 (Failed)
        940 - mgdyn_steady_quad_extruded_restart (Failed)
        941 - mgdyn_steady_quad_extruded_restart_np3 (Failed)
Errors while running CTest
Output from these tests are in: /root/build/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
make: *** [Makefile:91: test] Error 8

I will push the changes now.

[ettaka]

Nice work!

I tried to build
sudo singularity build --sandbox elmer elmer.def and it seems to work. However, I am not able to run (or find) the tests:

Singularity> cd /root/
Singularity> ls
go  snap

[Irvise]

Hi @ettaka how are you entering the Singularity environment? The files/tests should be located inside the container under /root/build. There, one should be able to run make test to execute them.

I am using the latest Apptainer version, but Singularity should also work...

[ettaka]

Hi @ettaka how are you entering the Singularity environment? The files/tests should be located inside the container under /root/build. There, one should be able to run make test to execute them.

I am using the latest Apptainer version, but Singularity should also work...

I used sudo singularity shell -w elmer
There was not a build directory in /root.

[Irvise]

Mmmmm... Maybe singularity is using the $USER directory? Could you check if, once in the singularity shell, there is a build directory under your $USER?

If not, I would encourage you to rebuild the "image" and look in the build log in the terminal for the directory that is being used during the build.

[Irvise]

Hi @ettaka, did you have any luck with the Apptainer file? If you want anything modified or more help feel free to ask!

Regards,
Fer