Mesh reader/writer with ADIOS2 #2618

jorgensd · 2023-04-11T14:16:26Z

This is the first step towards having function check-pointing in DOLFINx.

The code uses ADIOS2 as backend for reading and writing data.
No MPI communication required in either stage (until mesh creation/partitioning).

The following C++ code runs nicely

#include <dolfinx/common/MPI.h>
#include <dolfinx/io/ADIOS2Writers.h>
#include <dolfinx/mesh/Mesh.h>
#include <dolfinx/mesh/generation.h>
#include <filesystem>
#include <mpi.h>

using namespace dolfinx;

int main(int argc, char* argv[])
{
  dolfinx::init_logging(argc, argv);
  MPI_Init(&argc, &argv);

  {
    auto mesh
        = std::make_shared<mesh::Mesh<float>>(mesh::create_rectangle<float>(
            MPI_COMM_WORLD, {{{0.0, 0.0}, {1.0, 1.0}}}, {2, 3},
            mesh::CellType::triangle,
            mesh::create_cell_partitioner(mesh::GhostMode::none)));

#ifdef HAS_ADIOS2
    std::filesystem::__cxx11::path filename = "test";
    io::Checkpointer<float> outfile(mesh->comm(),
                                    filename.replace_extension("bp"), mesh);
    outfile.write(2.0);
    outfile.close();
    io::Checkpointer<float> infile(mesh->comm(),
                                   filename.replace_extension("bp"));
    mesh::Mesh<float> mesh_no_ghost
        = infile.read_mesh(2.0, mesh::GhostMode::none);
    mesh::Mesh<float> mesh_shared
        = infile.read_mesh(2.0, mesh::GhostMode::shared_facet);

    std::stringstream cc;
    const int rank = dolfinx::MPI::rank(MPI_COMM_WORLD);
    cc << "-" << rank << "-"
       << "\n";
    cc << "Mesh no ghosts: "
       << mesh_no_ghost.topology()
              ->index_map(mesh_no_ghost.topology()->dim())
              ->size_local()
       << " "
       << mesh_no_ghost.topology()
              ->index_map(mesh_no_ghost.topology()->dim())
              ->num_ghosts()
       << "\n";
    cc << "Mesh shared_facet: "
       << mesh_shared.topology()
              ->index_map(mesh_shared.topology()->dim())
              ->size_local()
       << " "
       << mesh_shared.topology()
              ->index_map(mesh_shared.topology()->dim())
              ->num_ghosts()
       << "\n";
    infile.close();

    io::Checkpointer<float> infile_self(MPI_COMM_SELF,
                                        filename.replace_extension("bp"));
    mesh::Mesh<float> mesh_self
        = infile_self.read_mesh(2.0, mesh::GhostMode::none);
    infile_self.close();
    cc << "MEsh self: "
       << mesh_self.topology()
              ->index_map(mesh_self.topology()->dim())
              ->size_local()
       << " "
       << mesh_self.topology()
              ->index_map(mesh_self.topology()->dim())
              ->num_ghosts()
       << "\n";
    std::cout << cc.str();

    io::VTXWriter<float> vtx_ng(
        mesh_no_ghost.comm(), "mesh_no_ghost.bp",
        std::make_shared<mesh::Mesh<float>>(mesh_no_ghost));
    vtx_ng.write(0.0);
    vtx_ng.close();

    io::VTXWriter<float> vtx_s(
        mesh_no_ghost.comm(), "mesh_shared.bp",
        std::make_shared<mesh::Mesh<float>>(mesh_shared));
    vtx_s.write(0.0);
    vtx_s.close();

    io::VTXWriter<float> vtx_self(
        mesh_self.comm(), "mesh" + std::to_string(rank) + ".bp",
        std::make_shared<mesh::Mesh<float>>(mesh_self));
    vtx_self.write(0.0);
    vtx_self.close();

#endif
  }
  MPI_Finalize();

  return 0;
}

 mpirun -n 3 ./build-dir-real/demo_interpolation-io 
-0-
Mesh no ghosts: 4 0
Mesh shared_facet: 4 2
MEsh self: 12 0
-1-
Mesh no ghosts: 4 0
Mesh shared_facet: 4 2
MEsh self: 12 0
-2-
Mesh no ghosts: 4 0
Mesh shared_facet: 4 4
MEsh self: 12 0

It would be good to get feedback on this design prior to implementing the python interface.

Once this is merged, I can start working on the function checkpointing, with the approach used in https://github.com/jorgensd/adios4dolfinx/blob/main/src/adios4dolfinx/checkpointing.py#L91-L173

garth-wells

Some preliminary comments.

Could this be divided into a 'read' PR, and a checkpoint PR after?

cpp/dolfinx/io/ADIOS2Writers.h

jorgensd · 2023-04-27T07:32:57Z

Some preliminary comments.

Could this be divided into a 'read' PR, and a checkpoint PR after?

No, as this reads the mesh from the mesh format this PR creates a writer for

cpp/dolfinx/io/ADIOS2Writers.h

jorgensd · 2023-05-02T08:00:37Z

Some preliminary comments.

Could this be divided into a 'read' PR, and a checkpoint PR after?

Changes have been made according to your suggestions, any further comments?

jorgensd · 2023-05-16T14:49:02Z

@garth-wells Any further comments?

garth-wells · 2023-06-25T11:59:52Z

@jorgensd could you make clearer in the PR comment what this changes does in practice?

garth-wells

Another round of comments.

Would be helpful to have clear statement in the PR of what functionality is being add.

cpp/dolfinx/io/ADIOS2Writers.h

garth-wells · 2023-06-25T12:06:31Z

cpp/dolfinx/io/ADIOS2Writers.h

+                                       int(geometry.cmaps()[0].variant()));
+  }
+
+  /// @brief Create a ADIOS2 checkpointer for reading a mesh from file


Does this need to be a class? Would a function suffice?

Maybe, it depends on how we want to do it when adding functions to the mix. have stand-alone functions for each of the operations, or having them interact in a class.

cpp/dolfinx/io/ADIOS2Writers.h

jorgensd · 2023-06-25T12:21:53Z

Another round of comments.

Would be helpful to have clear statement in the PR of what functionality is being add.

The code can write meshes to file on N processes, and read it back in on M processes. This is the first step towards having full checkpointing, as a next PR would add in writing/reading functions.

This is the reason for making the checkpointer a class, as one would write the mesh and function to the same file (and keep it alive during the process), avoiding duplicate initialization statements.

What differs in this code from the already existing Fides/VTXWriter is that:

the data is written to global arrays (so that they can be read in again on a different number of processes).
the topology is stored in DOLFINx order, i.e. the dofs are not permuted to VTK or GMSH ordering.

garth-wells · 2023-07-06T09:35:23Z

What differs in this code from the already existing Fides/VTXWriter is that:

the data is written to global arrays (so that they can be read in again on a different number of processes).

What's a 'global array'? An ADIOS2 concept?

The VTX files we write do have a local-to-global map array.

the topology is stored in DOLFINx order, i.e. the dofs are not permuted to VTK or GMSH ordering.

Do we want to guarantee that the DOLFINx remains unchanged? An advantage of VTK ordering is that we can freely change the internal DOLFINx ordering.

jorgensd · 2023-07-06T10:34:53Z

What's a 'global array'? An ADIOS2 concept?

Yes. Consider writing out a mesh with the VTXWriter in serial:

root@dokken-XPS-9320:~/shared/dolfinx/cpp/demo/poisson# ./build-dir-real64/demo_poisson 
root@dokken-XPS-9320:~/shared/dolfinx/cpp/demo/poisson# bpls -l mesh.bp/
  uint32_t  NumberOfCells        {1} = 1024 / 1024
  uint32_t  NumberOfNodes        {1} = 561 / 561
  int64_t   connectivity         [1]*{1024, 4} = 0 / 560
  double    geometry             [1]*{561, 3} = 0 / 2
  double    step                 scalar = 0
  uint32_t  types                scalar = 69
  uint8_t   vtkGhostType         [1]*{561} = 0 / 0
  int64_t   vtkOriginalPointIds  [1]*{561} = 0 / 560

is what you get in serial, but in parallel you get

root@dokken-XPS-9320:~/shared/dolfinx/cpp/demo/poisson# mpirun -n 4 ./build-dir-real64/demo_poisson 
root@dokken-XPS-9320:~/shared/dolfinx/cpp/demo/poisson# bpls -l mesh.bp/
  uint32_t  NumberOfCells        {4} = 272 / 288
  uint32_t  NumberOfNodes        {4} = 166 / 182
  int64_t   connectivity         [4]*{__, 4} = 0 / 181
  double    geometry             [4]*{__, 3} = 0 / 2
  double    step                 scalar = 0
  uint32_t  types                scalar = 69

which means that the data is split into block structures (equal to the number of processors).

This makes the data very fast to write, but very hard to read on any other number of processes than the one that wrote it.

A global array is what we are creating when we specify both local and global dimensions in put as well as the global offset, like:

adios2::Variable<T> local_geometry = impl_adios2::define_variable<T>(
      io, "geometry", {num_xdofs_global, gdim}, {local_range[0], 0},
      {num_xdofs_local, gdim});

which gives you a .bp file that looks like:

root@dokken-XPS-9320:~/shared/dolfinx/cpp/demo/checkpoint# mpirun -n 4 ./build-dir-real64/demo_interpolation-io 
root@dokken-XPS-9320:~/shared/dolfinx/cpp/demo/checkpoint# bpls -l test.bp/
  uint32_t  CellPermutations  {12} = 0 / 6
  int64_t   connectivity      {12, 3} = 0 / 11
  float     geometry          {12, 2} = 0 / 1
  double    step              scalar = 2

cpp/dolfinx/io/ADIOS2Writers.h

jhale · 2023-07-14T08:29:45Z

cpp/dolfinx/io/ADIOS2Writers.h

+  }
+
+  /// @brief Move constructor
+  Checkpointer(Checkpointer&& writer) = default;


jhale · 2023-07-14T08:31:27Z

cpp/dolfinx/io/ADIOS2Writers.h

+  ~Checkpointer() { close(); }
+
+  /// @brief Move assignment
+  Checkpointer& operator=(Checkpointer&& writer) = default;


cpp/dolfinx/io/ADIOS2Writers.h

…eckpointer

garth-wells

I think checkpointing needs a good high-level plan and discussion before merging. We need a summary of what we want to be able to do to inform the design. Parts of this PR mirror the VTX and Fides writers, but these designs are probably not ideal for different types.

When reading a file we don't know a priori what the types are. We need to able to (i) query this, or (ii) maybe return a std::variant.

garth-wells · 2023-07-17T07:31:28Z

cpp/dolfinx/io/ADIOS2Writers.h

+    std::span<const T> x = geometry.x();
+    std::size_t gdim = geometry.dim();
+
+    // Trim geometry to gdim and trunctate for owned entities only


This is a bit unusual for a 'checkpoint', which I would expect would align closely with the internal storage.

garth-wells · 2023-07-17T07:33:40Z

cpp/dolfinx/io/ADIOS2Writers.h

+  auto geometry_dmap = geometry.dofmap();
+
+  // Convert local geometry to global geometry
+  std::vector<std::int64_t> connectivity_out(num_cells_local


The terminology here is confusing. The variable name includes 'connectivity', but that data that gets copied in is geometry data.

garth-wells · 2023-07-17T07:34:32Z

cpp/dolfinx/io/ADIOS2Writers.h

+  engine.Put<std::int64_t>(local_topology, connectivity_out.data());
+
+  // Compute mesh permutations and store them to file. These are only
+  // needed for when we want to checkpoint functions.


'when we want to checkpoint fem::Functions.'

garth-wells · 2023-07-17T07:36:49Z

cpp/dolfinx/io/ADIOS2Writers.h

+
+/// Read and write mesh::Mesh objects with ADIOS2
+template <std::floating_point T>
+class Checkpointer


It doesn't seem logical that this class write and read - beyond the ADIOS2 handles the member data and methods are not shared across read and write. And the ADIOS2Writer class seems to already handle the ADISO2 handles.

garth-wells · 2023-07-17T07:42:22Z

cpp/dolfinx/io/ADIOS2Writers.h

+  std::unique_ptr<adios2::IO> _io;
+  std::unique_ptr<adios2::Engine> _engine;
+  std::shared_ptr<const mesh::Mesh<T>> _mesh;
+  U _u;


Is this used?

jorgensd added 2 commits April 11, 2023 09:11

Start adding mesh checkpointer work (writer works)

d69e060

Add both reader and writer.

1d44d9a

jorgensd requested review from michalhabera, garth-wells, chrisrichardson and IgorBaratta April 11, 2023 14:16

jorgensd and others added 8 commits April 11, 2023 14:17

Docs

b2a3be9

Fix docs

51fe43b

Merge branch 'main' into dokken/adios2-mesh-checkpointer

76583d9

Templating and type-casting

52eb516

Revert basix branch

1d5af28

Merge branch 'main' into dokken/adios2-mesh-checkpointer

2797c7d

Merge branch 'main' into dokken/adios2-mesh-checkpointer

359b38d

Merge branch 'main' into dokken/adios2-mesh-checkpointer

099cde1

garth-wells requested changes Apr 27, 2023

View reviewed changes

Various fixes to work with main branch

acd7f0f

jorgensd commented Apr 27, 2023

View reviewed changes

cpp/dolfinx/io/ADIOS2Writers.h Outdated Show resolved Hide resolved

jorgensd added 2 commits April 27, 2023 14:30

Apply suggestions from code review

6df8270

Merge branch 'main' into dokken/adios2-mesh-checkpointer

c5ec952

Merge branch 'main' into dokken/adios2-mesh-checkpointer

69222d1

Merge branch 'main' into dokken/adios2-mesh-checkpointer

33f5687

jorgensd requested a review from jhale June 1, 2023 06:43

jorgensd and others added 3 commits June 1, 2023 08:43

Merge branch 'main' into dokken/adios2-mesh-checkpointer

c179c92

Merge branch 'main' into dokken/adios2-mesh-checkpointer

095b1a7

Merge branch 'main' into dokken/adios2-mesh-checkpointer

16a4d71

garth-wells requested changes Jun 25, 2023

View reviewed changes

jorgensd linked an issue Jul 6, 2023 that may be closed by this pull request

Support reading of meshes from VTX/ADIOS2 files #2713

Open

Merge branch 'main' into dokken/adios2-mesh-checkpointer

05658d9

jorgensd and others added 7 commits July 6, 2023 10:35

Use read-random-access for checkpointer

aab0a11

Address Garths comments

66e544f

Merge branch 'main' into dokken/adios2-mesh-checkpointer

5bbb00e

Fix integer conversions

43e6297

More std::size_t

5efa5f4

Add docs

5b5a0cf

Merge branch 'main' into dokken/adios2-mesh-checkpointer

cd89b3f

jhale reviewed Jul 14, 2023

View reviewed changes

jorgensd and others added 7 commits July 14, 2023 08:36

Use static_cast

eefda05

Use auto for static cast

7df90c6

More auto

69fb40d

Merge branch 'main' into dokken/adios2-mesh-checkpointer

9541e0b

Update ADIOS2Writers.h

42a7097

Merge remote-tracking branch 'origin/main' into dokken/adios2-mesh-ch…

c83447c

…eckpointer

Formatting and style improvements

5b3d40d

garth-wells requested changes Jul 17, 2023

View reviewed changes

Merge branch 'main' into dokken/adios2-mesh-checkpointer

594fd28

RemDelaporteMathurin mentioned this pull request Oct 18, 2023

Support for XDMF export festim-dev/FESTIM#613

Merged

10 tasks

jorgensd closed this May 22, 2024

jorgensd deleted the dokken/adios2-mesh-checkpointer branch May 22, 2024 06:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mesh reader/writer with ADIOS2 #2618

Mesh reader/writer with ADIOS2 #2618

jorgensd commented Apr 11, 2023 •

edited

garth-wells left a comment

jorgensd commented Apr 27, 2023

jorgensd commented May 2, 2023

jorgensd commented May 16, 2023

garth-wells commented Jun 25, 2023

garth-wells left a comment

garth-wells Jun 25, 2023

jorgensd Jul 6, 2023

jorgensd commented Jun 25, 2023

garth-wells commented Jul 6, 2023

jorgensd commented Jul 6, 2023

jhale Jul 14, 2023

jhale Jul 14, 2023

garth-wells left a comment

garth-wells Jul 17, 2023

garth-wells Jul 17, 2023

garth-wells Jul 17, 2023

garth-wells Jul 17, 2023

garth-wells Jul 17, 2023

Mesh reader/writer with ADIOS2 #2618

Mesh reader/writer with ADIOS2 #2618

Conversation

jorgensd commented Apr 11, 2023 • edited

garth-wells left a comment

Choose a reason for hiding this comment

jorgensd commented Apr 27, 2023

jorgensd commented May 2, 2023

jorgensd commented May 16, 2023

garth-wells commented Jun 25, 2023

garth-wells left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorgensd commented Jun 25, 2023

garth-wells commented Jul 6, 2023

jorgensd commented Jul 6, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

garth-wells left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorgensd commented Apr 11, 2023 •

edited