Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

JSON/TOML backend: introduce abbreviated IO modes #1493

Open
wants to merge 25 commits into
base: dev
Choose a base branch
from
Open

JSON/TOML backend: introduce abbreviated IO modes #1493

wants to merge 25 commits into from

Conversation

franzpoeschel
Copy link
Contributor

@franzpoeschel franzpoeschel commented Aug 4, 2023 •
edited

Factored out of #1277, based on #1436

Introduces two options:

  1. json.dataset.mode = "template" (default: = "dataset"), write just the dataset extent and not the actual content
  2. json.attribute.mode = "short" (default: = "long"), write an attribute as {"software": "openPMD-api"} instead of {"software": {"value": "openPMD-api", "type": "STRING"}}

TODO:

  • Documentation
  • More testing: openpmd-pipe, erroring IO workflows – fixed, but add test
  • Merge TOML Backend #1436 first
  • In TOML: Enable abbreviated modes by default: Done, but document this
  • Maybe add an option to skip the 'Trying to write data to a template dataset. Will skip.' warning

Diff: https://github.com/franzpoeschel/openPMD-api/compare/parallel-json..topic-json-short-modes

@franzpoeschel franzpoeschel mentioned this pull request Aug 4, 2023
Open
15 tasks
github-advanced-security[bot]
github-advanced-security bot found potential problems Aug 4, 2023
View reviewed changes
src/IO/JSON/JSONIOHandlerImpl.cpp
Comment on lines +1258 to +1531
switch (j.type())
{
case nlohmann::json::value_t::null:
throw std::runtime_error(
"[JSON backend] Attribute must not be null: '" +
nameForErrorMessages + "'.");
case nlohmann::json::value_t::object:
throw std::runtime_error(
"[JSON backend] Shorthand-style attribute must not be an "
"object: '" +
nameForErrorMessages + "'.");
case nlohmann::json::value_t::array:
if (j.empty())
{
std::cerr << "Cannot recover datatype of empty vector without "
"explicit type annotation for attribute '"
<< nameForErrorMessages
<< "'. Will continue with VEC_INT datatype."
<< std::endl;
return std::vector<int>{};
}
else
{
auto valueType = j[0].type();
/*
* If the vector is of numeric type, it might happen that the
* first entry is an integer, but a later entry is a float.
* We need to pick the most generic datatype in that case.
*/
if (valueType == nlohmann::json::value_t::number_float ||
valueType == nlohmann::json::value_t::number_unsigned ||
valueType == nlohmann::json::value_t::number_integer)
{
valueType = unifyNumericType(j);
}
switch (valueType)
{
case nlohmann::json::value_t::null:
throw std::runtime_error(
"[JSON backend] Attribute must not be null: '" +
nameForErrorMessages + "'.");
case nlohmann::json::value_t::object:
throw std::runtime_error(
"[JSON backend] Invalid contained datatype (object) "
"inside vector-type attribute: '" +
nameForErrorMessages + "'.");
case nlohmann::json::value_t::array:
throw std::runtime_error(
"[JSON backend] Invalid contained datatype (array) "
"inside vector-type attribute: '" +
nameForErrorMessages + "'.");
case nlohmann::json::value_t::string:
return recoverVectorAttributeFromJson<std::string>(j);
case nlohmann::json::value_t::boolean:
throw std::runtime_error(
"[JSON backend] Attribute must not be vector of bool: "
"'" +
nameForErrorMessages + "'.");
case nlohmann::json::value_t::number_integer:
return recoverVectorAttributeFromJson<
nlohmann::json::number_integer_t>(j);
case nlohmann::json::value_t::number_unsigned:
return recoverVectorAttributeFromJson<
nlohmann::json::number_unsigned_t>(j);
case nlohmann::json::value_t::number_float:
return recoverVectorAttributeFromJson<
nlohmann::json::number_float_t>(j);
case nlohmann::json::value_t::binary:
throw std::runtime_error(
"[JSON backend] Attribute must not have binary type: "
"'" +
nameForErrorMessages + "'.");
case nlohmann::json::value_t::discarded:
throw std::runtime_error(
"Internal JSON parser datatype leaked into JSON "
"value.");
}
throw std::runtime_error("Unreachable!");
}
case nlohmann::json::value_t::string:
return j.get<std::string>();
case nlohmann::json::value_t::boolean:
return j.get<bool>();
case nlohmann::json::value_t::number_integer:
return j.get<nlohmann::json::number_integer_t>();
case nlohmann::json::value_t::number_unsigned:
return j.get<nlohmann::json::number_unsigned_t>();
case nlohmann::json::value_t::number_float:
return j.get<nlohmann::json::number_float_t>();
case nlohmann::json::value_t::binary:
throw std::runtime_error(
"[JSON backend] Attribute must not have binary type: '" +
nameForErrorMessages + "'.");
case nlohmann::json::value_t::discarded:
throw std::runtime_error(
"Internal JSON parser datatype leaked into JSON value.");
}

Check notice

Code scanning / CodeQL

Long switch case Note

Switch has at least one case that is too long:
array (67 lines)
.
test/SerialIOTest.cpp Fixed Show fixed Hide fixed
src/RecordComponent.cpp Fixed Show fixed Hide fixed
examples/14_toml_template.cpp
Comment on lines +69 to +98
// std::string config = R"(
// {
// "iteration_encoding": "variable_based",
// "toml": {
// "dataset": {"mode": "template"},
// "attribute": {"mode": "short"}
// }
// }
// )";

Check notice

Code scanning / CodeQL

Commented-out code Note

This comment appears to contain commented-out code.
@franzpoeschel franzpoeschel force-pushed the topic-json-short-modes branch 3 times, most recently from 4a969c5 to 7969864 Compare August 10, 2023 09:38
@franzpoeschel franzpoeschel mentioned this pull request Aug 17, 2023
Merged
4 tasks
@franzpoeschel
Copy link
Contributor Author

With this, the JSON backend is now sensibly usable from codes such as PIConGPU:

{
  "__openPMD_internal": {
    "attribute_mode": "short",
    "dataset_mode": "template"
  },
  "attributes": {
    "basePath": "/data/%T/",
    "date": "2023-10-11 09:57:53 +0200",
    "iterationEncoding": "fileBased",
    "iterationFormat": "simData_%06T",
    "meshesPath": "fields/",
    "openPMD": "1.1.0",
    "openPMDextension": 0,
    "particlesPath": "particles/",
    "picongpuIOVersionMajor": 2,
    "picongpuIOVersionMinor": 0,
    "software": "PIConGPU",
    "softwareVersion": "0.7.0-dev"
  },
  "data": {
    "100": {
      "attributes": {
        "cell_depth": 4.252342224121094,
        "cell_height": 1.0630855560302734,
        "cell_width": 4.252342224121094,
        "dt": 1,
        "eps0": 169.19711303710938,
        "iteration": 100,
        "mue0": 0.005910266190767288,
        "particleBoundary": [
          "absorbing",
          "absorbing",
          "absorbing",
          "absorbing",
          "absorbing",
          "absorbing"
        ],
        "particleBoundaryParameters": [
          "without field correction",
          "without field correction",
          "without field correction",
          "without field correction",
          "without field correction",
          "without field correction"
        ],
        "sim_slides": 0,
        "time": 100,
        "timeUnitSI": 1.39e-16,
        "unit_bfield": 40903.82224060171,
        "unit_charge": 1.1143237516482563e-15,
        "unit_efield": 12262657411105.05,
        "unit_energy": 5.694183860145225e-10,
        "unit_length": 4.1671151662e-08,
        "unit_mass": 6.335633991170193e-27,
        "unit_speed": 299792458,
        "unit_time": 1.39e-16
      },
      "fields": {
        "B": {
          "attributes": {
            "axisLabels": [
              "z",
              "y",
              "x"
            ],
            "dataOrder": "C",
            "fieldSmoothing": "none",
            "geometry": "cartesian",
            "gridGlobalOffset": [
              0,
              0,
              0
            ],
            "gridSpacing": [
              4.252342224121094,
              1.0630855560302734,
              4.252342224121094
            ],
            "gridUnitSI": 4.1671151662e-08,
            "timeOffset": 0,
            "unitDimension": [
              0,
              1,
              -2,
              -1,
              0,
              0,
              0
            ]
          },
          "x": {
            "attributes": {
              "position": [
                0,
                0.5,
                0.5
              ],
              "unitSI": 40903.82224060171
            },
            "datatype": "FLOAT",
            "extent": [
              32,
              32,
              32
            ]
          },
          "y": {
            "attributes": {
              "position": [
                0.5,
                0,
                0.5
              ],
              "unitSI": 40903.82224060171
            },
            "datatype": "FLOAT",
            "extent": [
              32,
              32,
              32
            ]
          },
          "z": {
            "attributes": {
              "position": [
                0.5,
                0.5,
                0
              ],
              "unitSI": 40903.82224060171
            },
            "datatype": "FLOAT",
            "extent": [
              32,
              32,
              32
            ]
          }
        },
        "E": {
          "attributes": {
            "axisLabels": [
              "z",
              "y",
              "x"
            ],
            "dataOrder": "C",
            "fieldSmoothing": "none",
            "geometry": "cartesian",
            "gridGlobalOffset": [
              0,
              0,
              0
            ],
            "gridSpacing": [
              4.252342224121094,
              1.0630855560302734,
              4.252342224121094
            ],
            "gridUnitSI": 4.1671151662e-08,
            "timeOffset": 0,
            "unitDimension": [
              1,
              1,
              -3,
              -1,
              0,
              0,
              0
            ]
          },
          "x": {
            "attributes": {
              "position": [
                0.5,
                0,
                0
              ],
              "unitSI": 12262657411105.05
            },
            "datatype": "FLOAT",
            "extent": [
              32,
              32,
              32
            ]
          },
          "y": {
            "attributes": {
              "position": [
                0,
                0.5,
                0
              ],
              "unitSI": 12262657411105.05
            },
            "datatype": "FLOAT",
            "extent": [
              32,
              32,
              32
            ]
          },
          "z": {
            "attributes": {
              "position": [
                0,
                0,
                0.5
              ],
              "unitSI": 12262657411105.05
            },
            "datatype": "FLOAT",
            "extent": [
              32,
              32,
              32
            ]
          }
        },
        "attributes": {
          "chargeCorrection": "none",
          "currentSmoothing": "none",
          "fieldBoundary": [
            "open",
            "open",
            "open",
            "open",
            "open",
            "open"
          ],
          "fieldBoundaryParameters": [
            "convolutional PML over 12 cells",
            "convolutional PML over 12 cells",
            "convolutional PML over 12 cells",
            "convolutional PML over 12 cells",
            "convolutional PML over 12 cells",
            "convolutional PML over 12 cells"
          ],
          "fieldSolver": "Yee"
        },
        "e_all_chargeDensity": {
          "attributes": {
            "axisLabels": [
              "z",
              "y",
              "x"
            ],
            "dataOrder": "C",
            "fieldSmoothing": "none",
            "geometry": "cartesian",
            "gridGlobalOffset": [
              0,
              0,
              0
            ],
            "gridSpacing": [
              4.252342224121094,
              1.0630855560302734,
              4.252342224121094
            ],
            "gridUnitSI": 4.1671151662e-08,
            "position": [
              0,
              0,
              0
            ],
            "timeOffset": 0,
            "unitDimension": [
              -3,
              0,
              1,
              1,
              0,
              0,
              0
            ],
            "unitSI": 15399438.226078901
          },
          "datatype": "FLOAT",
          "extent": [
            32,
            32,
            32
          ]
        },
        "e_all_energyDensity": {
          "attributes": {
            "axisLabels": [
              "z",
              "y",
              "x"
            ],
            "dataOrder": "C",
            "fieldSmoothing": "none",
            "geometry": "cartesian",
            "gridGlobalOffset": [
              0,
              0,
              0
            ],
            "gridSpacing": [
              4.252342224121094,
              1.0630855560302734,
              4.252342224121094
            ],
            "gridUnitSI": 4.1671151662e-08,
            "position": [
              0,
              0,
              0
            ],
            "timeOffset": 0,
            "unitDimension": [
              -1,
              1,
              -2,
              0,
              0,
              0,
              0
            ],
            "unitSI": 7869098408118.734
          },
          "datatype": "FLOAT",
          "extent": [
            32,
            32,
            32
          ]
        },
        "picongpu_idProvider": {
          "attributes": {
            "axisLabels": [
              "x"
            ],
            "dataOrder": "C",
            "geometry": "cartesian",
            "gridGlobalOffset": [
              0
            ],
            "gridSpacing": [
              1
            ],
            "gridUnitSI": 1,
            "timeOffset": 0,
            "unitDimension": [
              0,
              0,
              0,
              0,
              0,
              0,
              0
            ]
          },
          "nextId": {
            "attributes": {
              "position": [
                0
              ],
              "unitSI": 1
            },
            "datatype": "ULONG",
            "extent": [
              1,
              1,
              1
            ]
          },
          "startId": {
            "attributes": {
              "maxNumProc": 1,
              "position": [
                0
              ],
              "unitSI": 1
            },
            "datatype": "ULONG",
            "extent": [
              1,
              1,
              1
            ]
          }
        }
      },
      "particles": {
        "e": {
          "attributes": {
            "currentDeposition": "Esirkepov",
            "particleInterpolation": "uniform",
            "particlePush": "Boris",
            "particleShape": 2,
            "particleSmoothing": "none"
          },
          "charge": {
            "attributes": {
              "macroWeighted": 0,
              "shape": [
                0
              ],
              "timeOffset": 0,
              "unitDimension": [
                0,
                0,
                1,
                1,
                0,
                0,
                0
              ],
              "unitSI": 1.1143237516482563e-15,
              "value": -0.0001437801111023873,
              "weightingPower": 1
            }
          },
          "mass": {
            "attributes": {
              "macroWeighted": 0,
              "shape": [
                0
              ],
              "timeOffset": 0,
              "unitDimension": [
                0,
                1,
                0,
                0,
                0,
                0,
                0
              ],
              "unitSI": 6.335633991170193e-27,
              "value": 0.0001437801111023873,
              "weightingPower": 1
            }
          },
          "momentum": {
            "attributes": {
              "macroWeighted": 1,
              "timeOffset": 0,
              "unitDimension": [
                1,
                1,
                -1,
                0,
                0,
                0,
                0
              ],
              "weightingPower": 1
            },
            "x": {
              "attributes": {
                "shape": [
                  0
                ],
                "unitSI": 1.8993752872012626e-18,
                "value": 0
              }
            },
            "y": {
              "attributes": {
                "shape": [
                  0
                ],
                "unitSI": 1.8993752872012626e-18,
                "value": 0
              }
            },
            "z": {
              "attributes": {
                "shape": [
                  0
                ],
                "unitSI": 1.8993752872012626e-18,
                "value": 0
              }
            }
          },
          "particlePatches": {
            "extent": {
              "attributes": {
                "unitDimension": [
                  0,
                  0,
                  0,
                  0,
                  0,
                  0,
                  0
                ]
              },
              "x": {
                "attributes": {
                  "unitSI": 1
                },
                "datatype": "ULONG",
                "extent": [
                  1
                ]
              },
              "y": {
                "attributes": {
                  "unitSI": 1
                },
                "datatype": "ULONG",
                "extent": [
                  1
                ]
              },
              "z": {
                "attributes": {
                  "unitSI": 1
                },
                "datatype": "ULONG",
                "extent": [
                  1
                ]
              }
            },
            "numParticles": {
              "attributes": {
                "unitSI": 1
              },
              "datatype": "ULONG",
              "extent": [
                1
              ]
            },
            "numParticlesOffset": {
              "attributes": {
                "unitSI": 1
              },
              "datatype": "ULONG",
              "extent": [
                1
              ]
            },
            "offset": {
              "attributes": {
                "unitDimension": [
                  0,
                  0,
                  0,
                  0,
                  0,
                  0,
                  0
                ]
              },
              "x": {
                "attributes": {
                  "unitSI": 1
                },
                "datatype": "ULONG",
                "extent": [
                  1
                ]
              },
              "y": {
                "attributes": {
                  "unitSI": 1
                },
                "datatype": "ULONG",
                "extent": [
                  1
                ]
              },
              "z": {
                "attributes": {
                  "unitSI": 1
                },
                "datatype": "ULONG",
                "extent": [
                  1
                ]
              }
            }
          },
          "position": {
            "attributes": {
              "macroWeighted": 0,
              "timeOffset": 0,
              "unitDimension": [
                1,
                0,
                0,
                0,
                0,
                0,
                0
              ],
              "weightingPower": 0
            },
            "x": {
              "attributes": {
                "shape": [
                  0
                ],
                "unitSI": 1.7719999774007647e-07,
                "value": 0
              }
            },
            "y": {
              "attributes": {
                "shape": [
                  0
                ],
                "unitSI": 4.429999943501912e-08,
                "value": 0
              }
            },
            "z": {
              "attributes": {
                "shape": [
                  0
                ],
                "unitSI": 1.7719999774007647e-07,
                "value": 0
              }
            }
          },
          "positionOffset": {
            "attributes": {
              "macroWeighted": 0,
              "timeOffset": 0,
              "unitDimension": [
                1,
                0,
                0,
                0,
                0,
                0,
                0
              ],
              "weightingPower": 0
            },
            "x": {
              "attributes": {
                "shape": [
                  0
                ],
                "unitSI": 1.7719999774007647e-07,
                "value": 0
              }
            },
            "y": {
              "attributes": {
                "shape": [
                  0
                ],
                "unitSI": 4.429999943501912e-08,
                "value": 0
              }
            },
            "z": {
              "attributes": {
                "shape": [
                  0
                ],
                "unitSI": 1.7719999774007647e-07,
                "value": 0
              }
            }
          },
          "weighting": {
            "attributes": {
              "macroWeighted": 1,
              "shape": [
                0
              ],
              "timeOffset": 0,
              "unitDimension": [
                0,
                0,
                0,
                0,
                0,
                0,
                0
              ],
              "unitSI": 1,
              "value": 0,
              "weightingPower": 1
            }
          }
        }
      }
    }
  }
}

@franzpoeschel franzpoeschel force-pushed the topic-json-short-modes branch 3 times, most recently from c511364 to 544923b Compare October 12, 2023 09:39
@franzpoeschel franzpoeschel mentioned this pull request Oct 18, 2023
Open
5 tasks
@franzpoeschel franzpoeschel force-pushed the topic-json-short-modes branch 2 times, most recently from 8ca52da to 6464af8 Compare November 20, 2023 16:36
github-advanced-security[bot]
github-advanced-security bot found potential problems Nov 20, 2023
View reviewed changes
src/IO/JSON/JSONIOHandlerImpl.cpp Outdated
@@ -1269,40 +2009,165 @@
return (*obtainJsonContents(file))[filePosition->id];
}

void JSONIOHandlerImpl::putJsonContents(
auto JSONIOHandlerImpl::putJsonContents(

Check warning

Code scanning / CodeQL

Poorly documented large function Warning

Poorly documented function: fewer than 2% comments for a function of 160 lines.
@franzpoeschel franzpoeschel force-pushed the topic-json-short-modes branch 3 times, most recently from baf807e to 914aaaa Compare November 27, 2023 09:54
@franzpoeschel franzpoeschel force-pushed the topic-json-short-modes branch 3 times, most recently from 9269933 to 047e571 Compare December 22, 2023 18:35
franzpoeschel and others added 25 commits May 21, 2024 16:51
@franzpoeschel
Add openPMD 2.0 standard setting
ae2846e
@franzpoeschel
Fix clang-tidy warning
4340b36
@franzpoeschel
Introduce getStandardMaximum(), deprecate getStandard()
deee96e
@franzpoeschel
Add warning: openPMD 2.0 still under development
74b90f6
@franzpoeschel
Introduce dataset template mode to JSON backend
f76760b
@franzpoeschel
Write used mode to JSON file
352ef3a
@franzpoeschel
Use Attribute::getOptional for snapshot attribute
a171379
@franzpoeschel
Introduce attribute mode
299f206
@franzpoeschel
Add example 14_toml_template.cpp
8791828
@franzpoeschel
Use Datatype::UNDEFINED to indicate no dataset definition in template
82244b4
@franzpoeschel
Extend example
acf7f4c
@franzpoeschel
Test short attribute mode
98d133c
@franzpoeschel
Copy datatypeToString to JSON implementation
3de7660
@franzpoeschel
Fix after rebase: Init JSON config in parallel mode
fc05946
@franzpoeschel
Fix after rebase: Don't erase JSON datasets when writing
0d62f1c
@franzpoeschel
openpmd-pipe: use short modes for test
db9de60
@franzpoeschel
Less intrusive warnings, allow disabling them
e28e176
@franzpoeschel
TOML: Use short modes by default
52c5d85
@franzpoeschel
Python formatting
3199c4c
@pre-commit-ci @franzpoeschel
[pre-commit.ci] auto fixes from pre-commit.com hooks
9d89ef5 
for more information, see https://pre-commit.ci
@franzpoeschel
Documentation
f2a313f
@franzpoeschel
Short mode in default in openPMD >= 2.
693d401
@franzpoeschel
Short value by default in TOML
661acb1
@franzpoeschel
Store the openPMD version information in the IOHandler
9032917
@franzpoeschel
Fixes
7afed41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
backend: JSON
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
1 participant
@franzpoeschel