Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Problems with reading BP5 output #1457

Open
pordyna opened this issue Jun 6, 2023 · 18 comments
Open

Problems with reading BP5 output #1457

pordyna opened this issue Jun 6, 2023 · 18 comments
Labels
backend: ADIOS2 bug

Comments

@pordyna
Copy link
Contributor

pordyna commented Jun 6, 2023

Describe the bug
Process gets killed when opening an ADIOS BP5 series generated with PIConGPU in READ_ONLY mode. I generated a fairly large openPMD dataset (107TiB) with the following PIConGPU IO configuration:

toml

# The following parameters need not be specified
# If a parameter is left unspecified, it falls back to its default value
file = "simData"    # replaces --openPMD.file,
                   # given value is the default

infix = ""         # replaces --openPMD.infix,
                   # default is "%06T"
ext = "bp5"         # replaces --openPMD.ext,
                   # given value is the default
backend_config = "@../input/etc/picongpu/adios_config.json"    # replaces --openPMD.json,
                                           # default is "{}"
data_preparation_strategy = "mappedMemory" # replaces --openPMD.dataPreparationStrategy,
                                           # default is "doubleBuffer"


# Periods and data sources are specified independently per reading application
# The application names can be arbitrary and are not interpreted, except
# potentially for logging and other messages.
[sink.disk.period]
# Each entry here denotes a periodicity combined with data sources requested
# by the reading code from PIConGPU at the specified periodicity
"4356:4356,450" = "species_all"
"1:1,252" = "fields_all"
"0:0,4356:9072:2" = ["e_all_density", "e_all_densityOverGammaSquared", "B"]

adios_config.json

{"adios2": {                                       
    "engine": {                                 
        "parameters": {                         
             "AggregatorRatio" : "1"            
             ,"BufferChunkSize" : "2147381248"  
        }                                           
    }                                         
    , "dataset": {                            
        "operators": [ {                        
            "type": "blosc"                     
            , "parameters": {                     
                "clevel": "1"                     
                , "compressor": "zstd"            
                , "doshuffle": "BLOSC_BITSHUFFLE" 
            }                                       
        } ]                                       
    }                                             
}}

openpd-ls simData.bp5, bpls simData.bp5, as well as io.Series("<...>/simData.bp5", io.Access.read_only) crash and go out of memory.
Using io.Series("<...>/simData.bp5", io.Access.read_only), {\"defer_iteration_parsing\": true}") doesn't help, but switching to io.Access.read_linear does. It looks like the series was written with variable encoding.

Executing the following script with the memory profiler memray shows that the ADIOS reader is allocating a gigantic buffer somewhere in readGorVBased. The full stack with the allocation sizes can be found in the attached html file memray-flamegraph-test_open_series.py.20346 .html.txt.

from pathlib import Path
import openpmd_api as io
base_path = Path(
![newplot](https://github.com/openPMD/openPMD-api/assets/41141557/31fc9979-5487-4ee9-8758-aa420428709a)
"/global/cfs/cdirs/m4251/pordyna/runs/hydrogen_toy_density/3D/2023_06_02_N04_final")
path = base_path / 'simOutput/openPMD/simData.bp5'
series = io.Series(str(path), io.Access.read_only,  "{\"defer_iteration_parsing\": true}")
for attr in series.attributes:
    print(f"{attr}: {series.get_attribute(attr)}")

newplot

Expected behavior and some questions

  • openpmd-ls and bpls should work even with variable encoding
  • Is this out of memory crash already expected due to the known limitations of using io.Access.read_only with variable encoding, or is it an extra bug?
  • Is it the desired behavior that using .bp5 together with no infix (like %06T) creates a series with variable encoding even though it does not support random access? Is this an openpmd-api default or is it set in picongpu @franzpoeschel?
  • Can I somehow use BP5 with group based encoding? I was trying to avoid using file based encoding with so meany iterations.

Software Environment

  • version of openPMD-api: 0.15.1
  • installed openPMD-api via: from source (using the python package compiled together with the api for running picongpu)
  • operating system: [name and version]
  • machine: perlmutter
  • name and version of Python implementation: CPython 3.9.16
  • version of HDF5: 1.14.0
  • version of ADIOS2: 2.9.0
  • name and version of MPI: mpich 8.1.25 (3.1 standard version)
@pordyna pordyna added the bug label Jun 6, 2023
@franzpoeschel
Copy link
Contributor

Do you get any output for something like bpls -t <dataset.bp5> | head -n 200? If read_linear works, then that should work, too.

From your configuration, you create a group-based Series. Even for a group-based Series, PIConGPU will write the output to multiple IO steps in ADIOS2. That is necessary because otherwise a simulation crash would mean that all data is lost, and in all engines except BP5 you would run out of memory without closing a step at some point.

If the bpls command above works, I expect that you will see that the group-based encoding creates a new set of variables and attributes in ADIOS2 for each step, e.g.

/data/0/meshes/E/x
...
/data/100/meshes/E/x
...
/data/200/meshes/E/x
...

To confirm an unwieldy size of metadata, what is the output of something like ls -lish <dataset.bp5>? If their size is somewhat manageable, can you upload the folder as a zip (all files except for the large data.* files)?

Looking at the memory trace, it seem like something in the BP5 parser scales really badly with a large set of such metadata, likely a bug.

Variable-based encoding would be a solution here, replacing the above with a single variable /data/meshes/E/x that is reused along the steps. But the variable-based encoding requires a new functionality in the ADIOS2 backend, and we will soon replace the current experimental implementation of that functionality with something entirely different.
However, while variable-based encoding avoids this buildup of metadata, it requires read_linear for reading anyway..

Side note: Blosc in BP5 is currently broken, writing will succeed, but you might get unreadable datasets.

@pordyna
Copy link
Contributor Author

pordyna commented Jun 7, 2023

@franzpoeschel This is the output of bpls -t, writing this out is fairly slow, but maybe it is to be expected. I will do some profiling with read_lin as well.

$ bpls -t simData.bp5/ | head -n 200
Step 0:
  float     /data/0/fields/B/x                                      {1440, 1440, 1440}
  float     /data/0/fields/B/y                                      {1440, 1440, 1440}
  float     /data/0/fields/B/z                                      {1440, 1440, 1440}
  float     /data/0/fields/E/x                                      {1440, 1440, 1440}
  float     /data/0/fields/E/y                                      {1440, 1440, 1440}
  float     /data/0/fields/E/z                                      {1440, 1440, 1440}
  float     /data/0/fields/H_all_density                            {1440, 1440, 1440}
  float     /data/0/fields/H_all_energyDensity                      {1440, 1440, 1440}
  float     /data/0/fields/H_all_macroParticleCounter               {1440, 1440, 1440}
  float     /data/0/fields/J/x                                      {1440, 1440, 1440}
  float     /data/0/fields/J/y                                      {1440, 1440, 1440}
  float     /data/0/fields/J/z                                      {1440, 1440, 1440}
  float     /data/0/fields/e_all_density                            {1440, 1440, 1440}
  float     /data/0/fields/e_all_densityOverGammaSquared            {1440, 1440, 1440}
  float     /data/0/fields/e_all_energyDensity                      {1440, 1440, 1440}
  float     /data/0/fields/e_all_macroParticleCounter               {1440, 1440, 1440}
  uint64_t  /data/0/fields/picongpu_idProvider/nextId               {12, 6, 6}
  uint64_t  /data/0/fields/picongpu_idProvider/startId              {12, 6, 6}
  uint64_t  /data/0/particles/H/id                                  {12122999040}
  float     /data/0/particles/H/momentum/x                          {12122999040}
  float     /data/0/particles/H/momentum/y                          {12122999040}
  float     /data/0/particles/H/momentum/z                          {12122999040}
  uint64_t  /data/0/particles/H/particlePatches/extent/x            {432}
  uint64_t  /data/0/particles/H/particlePatches/extent/y            {432}
  uint64_t  /data/0/particles/H/particlePatches/extent/z            {432}
  uint64_t  /data/0/particles/H/particlePatches/numParticles        {432}
  uint64_t  /data/0/particles/H/particlePatches/numParticlesOffset  {432}
  uint64_t  /data/0/particles/H/particlePatches/offset/x            {432}
  uint64_t  /data/0/particles/H/particlePatches/offset/y            {432}
  uint64_t  /data/0/particles/H/particlePatches/offset/z            {432}
  float     /data/0/particles/H/position/x                          {12122999040}
  float     /data/0/particles/H/position/y                          {12122999040}
  float     /data/0/particles/H/position/z                          {12122999040}
  int32_t   /data/0/particles/H/positionOffset/x                    {12122999040}
  int32_t   /data/0/particles/H/positionOffset/y                    {12122999040}
  int32_t   /data/0/particles/H/positionOffset/z                    {12122999040}
  float     /data/0/particles/H/weighting                           {12122999040}
  uint64_t  /data/0/particles/e/id                                  {12122999040}
  float     /data/0/particles/e/momentum/x                          {12122999040}
  float     /data/0/particles/e/momentum/y                          {12122999040}
  float     /data/0/particles/e/momentum/z                          {12122999040}
  uint64_t  /data/0/particles/e/particlePatches/extent/x            {432}
  uint64_t  /data/0/particles/e/particlePatches/extent/y            {432}
  uint64_t  /data/0/particles/e/particlePatches/extent/z            {432}
  uint64_t  /data/0/particles/e/particlePatches/numParticles        {432}
  uint64_t  /data/0/particles/e/particlePatches/numParticlesOffset  {432}
  uint64_t  /data/0/particles/e/particlePatches/offset/x            {432}
  uint64_t  /data/0/particles/e/particlePatches/offset/y            {432}
  uint64_t  /data/0/particles/e/particlePatches/offset/z            {432}
  float     /data/0/particles/e/position/x                          {12122999040}
  float     /data/0/particles/e/position/y                          {12122999040}
  float     /data/0/particles/e/position/z                          {12122999040}
  int32_t   /data/0/particles/e/positionOffset/x                    {12122999040}
  int32_t   /data/0/particles/e/positionOffset/y                    {12122999040}
  int32_t   /data/0/particles/e/positionOffset/z                    {12122999040}
  float     /data/0/particles/e/weighting                           {12122999040}
Step 1:
  float     /data/1/fields/B/x                            {1440, 1440, 1440}
  float     /data/1/fields/B/y                            {1440, 1440, 1440}
  float     /data/1/fields/B/z                            {1440, 1440, 1440}
  float     /data/1/fields/E/x                            {1440, 1440, 1440}
  float     /data/1/fields/E/y                            {1440, 1440, 1440}
  float     /data/1/fields/E/z                            {1440, 1440, 1440}
  float     /data/1/fields/H_all_density                  {1440, 1440, 1440}
  float     /data/1/fields/H_all_energyDensity            {1440, 1440, 1440}
  float     /data/1/fields/H_all_macroParticleCounter     {1440, 1440, 1440}
  float     /data/1/fields/J/x                            {1440, 1440, 1440}
  float     /data/1/fields/J/y                            {1440, 1440, 1440}
  float     /data/1/fields/J/z                            {1440, 1440, 1440}
  float     /data/1/fields/e_all_density                  {1440, 1440, 1440}
  float     /data/1/fields/e_all_densityOverGammaSquared  {1440, 1440, 1440}
  float     /data/1/fields/e_all_energyDensity            {1440, 1440, 1440}
  float     /data/1/fields/e_all_macroParticleCounter     {1440, 1440, 1440}
  uint64_t  /data/1/fields/picongpu_idProvider/nextId     {12, 6, 6}
  uint64_t  /data/1/fields/picongpu_idProvider/startId    {12, 6, 6}
Step 2:
  float     /data/252/fields/B/x                            {1440, 1440, 1440}
  float     /data/252/fields/B/y                            {1440, 1440, 1440}
  float     /data/252/fields/B/z                            {1440, 1440, 1440}
  float     /data/252/fields/E/x                            {1440, 1440, 1440}
  float     /data/252/fields/E/y                            {1440, 1440, 1440}
  float     /data/252/fields/E/z                            {1440, 1440, 1440}
  float     /data/252/fields/H_all_density                  {1440, 1440, 1440}
  float     /data/252/fields/H_all_energyDensity            {1440, 1440, 1440}
  float     /data/252/fields/H_all_macroParticleCounter     {1440, 1440, 1440}
  float     /data/252/fields/J/x                            {1440, 1440, 1440}
  float     /data/252/fields/J/y                            {1440, 1440, 1440}
  float     /data/252/fields/J/z                            {1440, 1440, 1440}
  float     /data/252/fields/e_all_density                  {1440, 1440, 1440}
  float     /data/252/fields/e_all_densityOverGammaSquared  {1440, 1440, 1440}
  float     /data/252/fields/e_all_energyDensity            {1440, 1440, 1440}
  float     /data/252/fields/e_all_macroParticleCounter     {1440, 1440, 1440}
  uint64_t  /data/252/fields/picongpu_idProvider/nextId     {12, 6, 6}
  uint64_t  /data/252/fields/picongpu_idProvider/startId    {12, 6, 6}
Step 3:
  uint64_t  /data/450/fields/picongpu_idProvider/nextId               {12, 6, 6}
  uint64_t  /data/450/fields/picongpu_idProvider/startId              {12, 6, 6}
  uint64_t  /data/450/particles/H/id                                  {12122999040}
  float     /data/450/particles/H/momentum/x                          {12122999040}
  float     /data/450/particles/H/momentum/y                          {12122999040}
  float     /data/450/particles/H/momentum/z                          {12122999040}
  uint64_t  /data/450/particles/H/particlePatches/extent/x            {432}
  uint64_t  /data/450/particles/H/particlePatches/extent/y            {432}
  uint64_t  /data/450/particles/H/particlePatches/extent/z            {432}
  uint64_t  /data/450/particles/H/particlePatches/numParticles        {432}
  uint64_t  /data/450/particles/H/particlePatches/numParticlesOffset  {432}
  uint64_t  /data/450/particles/H/particlePatches/offset/x            {432}
  uint64_t  /data/450/particles/H/particlePatches/offset/y            {432}
  uint64_t  /data/450/particles/H/particlePatches/offset/z            {432}
  float     /data/450/particles/H/position/x                          {12122999040}
  float     /data/450/particles/H/position/y                          {12122999040}
  float     /data/450/particles/H/position/z                          {12122999040}
  int32_t   /data/450/particles/H/positionOffset/x                    {12122999040}
  int32_t   /data/450/particles/H/positionOffset/y                    {12122999040}
  int32_t   /data/450/particles/H/positionOffset/z                    {12122999040}
  float     /data/450/particles/H/weighting                           {12122999040}
  uint64_t  /data/450/particles/e/id                                  {12122999040}
  float     /data/450/particles/e/momentum/x                          {12122999040}
  float     /data/450/particles/e/momentum/y                          {12122999040}
  float     /data/450/particles/e/momentum/z                          {12122999040}
  uint64_t  /data/450/particles/e/particlePatches/extent/x            {432}
  uint64_t  /data/450/particles/e/particlePatches/extent/y            {432}
  uint64_t  /data/450/particles/e/particlePatches/extent/z            {432}
  uint64_t  /data/450/particles/e/particlePatches/numParticles        {432}
  uint64_t  /data/450/particles/e/particlePatches/numParticlesOffset  {432}
  uint64_t  /data/450/particles/e/particlePatches/offset/x            {432}
  uint64_t  /data/450/particles/e/particlePatches/offset/y            {432}
  uint64_t  /data/450/particles/e/particlePatches/offset/z            {432}
  float     /data/450/particles/e/position/x                          {12122999040}
  float     /data/450/particles/e/position/y                          {12122999040}
  float     /data/450/particles/e/position/z                          {12122999040}
  int32_t   /data/450/particles/e/positionOffset/x                    {12122999040}
  int32_t   /data/450/particles/e/positionOffset/y                    {12122999040}
  int32_t   /data/450/particles/e/positionOffset/z                    {12122999040}
  float     /data/450/particles/e/weighting                           {12122999040}
Step 4:
  float     /data/504/fields/B/x                            {1440, 1440, 1440}
  float     /data/504/fields/B/y                            {1440, 1440, 1440}
  float     /data/504/fields/B/z                            {1440, 1440, 1440}
  float     /data/504/fields/E/x                            {1440, 1440, 1440}
  float     /data/504/fields/E/y                            {1440, 1440, 1440}
  float     /data/504/fields/E/z                            {1440, 1440, 1440}
  float     /data/504/fields/H_all_density                  {1440, 1440, 1440}
  float     /data/504/fields/H_all_energyDensity            {1440, 1440, 1440}
  float     /data/504/fields/H_all_macroParticleCounter     {1440, 1440, 1440}
  float     /data/504/fields/J/x                            {1440, 1440, 1440}
  float     /data/504/fields/J/y                            {1440, 1440, 1440}
  float     /data/504/fields/J/z                            {1440, 1440, 1440}
  float     /data/504/fields/e_all_density                  {1440, 1440, 1440}
  float     /data/504/fields/e_all_densityOverGammaSquared  {1440, 1440, 1440}
  float     /data/504/fields/e_all_energyDensity            {1440, 1440, 1440}
  float     /data/504/fields/e_all_macroParticleCounter     {1440, 1440, 1440}
  uint64_t  /data/504/fields/picongpu_idProvider/nextId     {12, 6, 6}
  uint64_t  /data/504/fields/picongpu_idProvider/startId    {12, 6, 6}
Step 5:
  float     /data/756/fields/B/x                            {1440, 1440, 1440}
  float     /data/756/fields/B/y                            {1440, 1440, 1440}
  float     /data/756/fields/B/z                            {1440, 1440, 1440}
  float     /data/756/fields/E/x                            {1440, 1440, 1440}
  float     /data/756/fields/E/y                            {1440, 1440, 1440}
  float     /data/756/fields/E/z                            {1440, 1440, 1440}
  float     /data/756/fields/H_all_density                  {1440, 1440, 1440}
  float     /data/756/fields/H_all_energyDensity            {1440, 1440, 1440}
  float     /data/756/fields/H_all_macroParticleCounter     {1440, 1440, 1440}
  float     /data/756/fields/J/x                            {1440, 1440, 1440}
  float     /data/756/fields/J/y                            {1440, 1440, 1440}
  float     /data/756/fields/J/z                            {1440, 1440, 1440}
  float     /data/756/fields/e_all_density                  {1440, 1440, 1440}
  float     /data/756/fields/e_all_densityOverGammaSquared  {1440, 1440, 1440}
  float     /data/756/fields/e_all_energyDensity            {1440, 1440, 1440}
  float     /data/756/fields/e_all_macroParticleCounter     {1440, 1440, 1440}
  uint64_t  /data/756/fields/picongpu_idProvider/nextId     {12, 6, 6}
  uint64_t  /data/756/fields/picongpu_idProvider/startId    {12, 6, 6}
Step 6:
  uint64_t  /data/900/fields/picongpu_idProvider/nextId               {12, 6, 6}
  uint64_t  /data/900/fields/picongpu_idProvider/startId              {12, 6, 6}
  uint64_t  /data/900/particles/H/id                                  {12122999040}
  float     /data/900/particles/H/momentum/x                          {12122999040}
  float     /data/900/particles/H/momentum/y                          {12122999040}
  float     /data/900/particles/H/momentum/z                          {12122999040}
  uint64_t  /data/900/particles/H/particlePatches/extent/x            {432}
  uint64_t  /data/900/particles/H/particlePatches/extent/y            {432}
  uint64_t  /data/900/particles/H/particlePatches/extent/z            {432}
  uint64_t  /data/900/particles/H/particlePatches/numParticles        {432}
  uint64_t  /data/900/particles/H/particlePatches/numParticlesOffset  {432}
  uint64_t  /data/900/particles/H/particlePatches/offset/x            {432}
  uint64_t  /data/900/particles/H/particlePatches/offset/y            {432}
  uint64_t  /data/900/particles/H/particlePatches/offset/z            {432}
  float     /data/900/particles/H/position/x                          {12122999040}
  float     /data/900/particles/H/position/y                          {12122999040}
  float     /data/900/particles/H/position/z                          {12122999040}
  int32_t   /data/900/particles/H/positionOffset/x                    {12122999040}
  int32_t   /data/900/particles/H/positionOffset/y                    {12122999040}
  int32_t   /data/900/particles/H/positionOffset/z                    {12122999040}
  float     /data/900/particles/H/weighting                           {12122999040}
  uint64_t  /data/900/particles/e/id                                  {12122998934}
  float     /data/900/particles/e/momentum/x                          {12122998934}
  float     /data/900/particles/e/momentum/y                          {12122998934}
  float     /data/900/particles/e/momentum/z                          {12122998934}

@franzpoeschel
Copy link
Contributor

Thanks, this confirms that you are using the group-based encoding.

Can you also do this, please:

To confirm an unwieldy size of metadata, what is the output of something like ls -lish <dataset.bp5>? If their size is somewhat manageable, can you upload the folder as a zip (all files except for the large data.* files)?

An example ls output would be something like:

$ ls openPMD/simData_900.bp5/ -lish
total 2.3M
179700456 2.3M -rw-r--r-- 1 franzpoeschel franzpoeschel 2.3M Jun  5 18:16 data.0
179700458  12K -rw-r--r-- 1 franzpoeschel franzpoeschel 9.7K Jun  5 18:16 md.0
179700459 4.0K -rw-r--r-- 1 franzpoeschel franzpoeschel  626 Jun  5 18:16 md.idx
179700457 4.0K -rw-r--r-- 1 franzpoeschel franzpoeschel 2.6K Jun  5 18:16 mmd.0
179700460 4.0K -rw-r--r-- 1 franzpoeschel franzpoeschel  534 Jun  5 18:16 profiling.json

The three files md.0, md.idx, mmd.0 are those that contain the metadata and which are likely the root of the problem.

@pordyna
Copy link
Contributor Author

pordyna commented Jun 7, 2023 •
edited

@franzpoeschel, well md.0 is more or less 0.5TB

$ total 107T
4663451215 815G -rw-r----- 1 pordyna pordyna 815G Jun  3 02:17 data.0
4663451217 861G -rw-r----- 1 pordyna pordyna 861G Jun  3 02:17 data.1
4662707675 860G -rw-r----- 1 pordyna pordyna 860G Jun  3 02:17 data.10
4663089329 120G -rw-r----- 1 pordyna pordyna 120G Jun  3 02:17 data.100
4663352797 120G -rw-r----- 1 pordyna pordyna 120G Jun  3 02:17 data.101
4663089914 120G -rw-r----- 1 pordyna pordyna 120G Jun  3 02:17 data.102
4663352798 121G -rw-r----- 1 pordyna pordyna 121G Jun  3 02:17 data.103
3082640322 120G -rw-r----- 1 pordyna pordyna 120G Jun  3 02:17 data.104
3082640343 118G -rw-r----- 1 pordyna pordyna 118G Jun  3 02:17 data.105
3082640305 116G -rw-r----- 1 pordyna pordyna 116G Jun  3 02:17 data.106
3082902675 105G -rw-r----- 1 pordyna pordyna 105G Jun  3 02:17 data.107
4662707660 104G -rw-r----- 1 pordyna pordyna 104G Jun  3 02:17 data.108
4662707669 116G -rw-r----- 1 pordyna pordyna 116G Jun  3 02:17 data.109
4662707673 816G -rw-r----- 1 pordyna pordyna 816G Jun  3 02:17 data.11
4662707664 118G -rw-r----- 1 pordyna pordyna 118G Jun  3 02:17 data.110
4662707671 119G -rw-r----- 1 pordyna pordyna 119G Jun  3 02:17 data.111
4662700467 118G -rw-r----- 1 pordyna pordyna 118G Jun  3 02:17 data.112
4662700471 118G -rw-r----- 1 pordyna pordyna 118G Jun  3 02:17 data.113
4662700465 118G -rw-r----- 1 pordyna pordyna 118G Jun  3 02:17 data.114
4662700472 118G -rw-r----- 1 pordyna pordyna 118G Jun  3 02:17 data.115
4662481283 119G -rw-r----- 1 pordyna pordyna 119G Jun  3 02:17 data.116
4662481279 118G -rw-r----- 1 pordyna pordyna 118G Jun  3 02:17 data.117
4662481280 116G -rw-r----- 1 pordyna pordyna 116G Jun  3 02:17 data.118
4662481282 104G -rw-r----- 1 pordyna pordyna 104G Jun  3 02:17 data.119
4662658166 203G -rw-r----- 1 pordyna pordyna 203G Jun  3 02:17 data.12
4650052702  86G -rw-r----- 1 pordyna pordyna  86G Jun  3 02:17 data.120
4650052840  93G -rw-r----- 1 pordyna pordyna  93G Jun  3 02:17 data.121
4650052713  92G -rw-r----- 1 pordyna pordyna  92G Jun  3 02:17 data.122
4650052664  91G -rw-r----- 1 pordyna pordyna  91G Jun  3 02:17 data.123
4662824922  90G -rw-r----- 1 pordyna pordyna  90G Jun  3 02:17 data.124
4662824932  90G -rw-r----- 1 pordyna pordyna  90G Jun  3 02:17 data.125
4662824933  90G -rw-r----- 1 pordyna pordyna  90G Jun  3 02:17 data.126
4662824917  90G -rw-r----- 1 pordyna pordyna  90G Jun  3 02:17 data.127
4658520234  92G -rw-r----- 1 pordyna pordyna  92G Jun  3 02:17 data.128
4658257273  93G -rw-r----- 1 pordyna pordyna  93G Jun  3 02:17 data.129
4662658167 218G -rw-r----- 1 pordyna pordyna 218G Jun  3 02:17 data.13
4658257292  93G -rw-r----- 1 pordyna pordyna  93G Jun  3 02:17 data.130
4657994410  86G -rw-r----- 1 pordyna pordyna  86G Jun  3 02:17 data.131
4663140462 217G -rw-r----- 1 pordyna pordyna 217G Jun  3 02:17 data.132
4663140464 233G -rw-r----- 1 pordyna pordyna 233G Jun  3 02:17 data.133
4663140465 238G -rw-r----- 1 pordyna pordyna 238G Jun  3 02:17 data.134
4663140463 240G -rw-r----- 1 pordyna pordyna 240G Jun  3 02:17 data.135
2561465776 244G -rw-r----- 1 pordyna pordyna 244G Jun  3 02:17 data.136
2561465778 249G -rw-r----- 1 pordyna pordyna 249G Jun  3 02:17 data.137
2561465786 249G -rw-r----- 1 pordyna pordyna 249G Jun  3 02:17 data.138
2561465782 245G -rw-r----- 1 pordyna pordyna 245G Jun  3 02:17 data.139
4662658165 225G -rw-r----- 1 pordyna pordyna 225G Jun  3 02:17 data.14
4666501729 241G -rw-r----- 1 pordyna pordyna 241G Jun  3 02:17 data.140
4662817643 238G -rw-r----- 1 pordyna pordyna 238G Jun  3 02:17 data.141
4666501730 233G -rw-r----- 1 pordyna pordyna 233G Jun  3 02:17 data.142
4666501690 217G -rw-r----- 1 pordyna pordyna 217G Jun  3 02:17 data.143
4666583352 220G -rw-r----- 1 pordyna pordyna 220G Jun  3 02:17 data.144
4666583355 230G -rw-r----- 1 pordyna pordyna 230G Jun  3 02:17 data.145
4666583353 230G -rw-r----- 1 pordyna pordyna 230G Jun  3 02:17 data.146
4666583354 229G -rw-r----- 1 pordyna pordyna 229G Jun  3 02:17 data.147
4664941030 229G -rw-r----- 1 pordyna pordyna 229G Jun  3 02:17 data.148
4664941032 230G -rw-r----- 1 pordyna pordyna 230G Jun  3 02:17 data.149
4662658168 233G -rw-r----- 1 pordyna pordyna 233G Jun  3 02:17 data.15
4664941031 230G -rw-r----- 1 pordyna pordyna 230G Jun  3 02:17 data.150
4664940898 228G -rw-r----- 1 pordyna pordyna 228G Jun  3 02:17 data.151
4662530100 229G -rw-r----- 1 pordyna pordyna 229G Jun  3 02:17 data.152
4662530110 230G -rw-r----- 1 pordyna pordyna 230G Jun  3 02:17 data.153
4662530101 230G -rw-r----- 1 pordyna pordyna 230G Jun  3 02:17 data.154
4662530078 221G -rw-r----- 1 pordyna pordyna 221G Jun  3 02:17 data.155
4662529097 106G -rw-r----- 1 pordyna pordyna 106G Jun  3 02:17 data.156
4662529096 118G -rw-r----- 1 pordyna pordyna 118G Jun  3 02:17 data.157
4662529092 121G -rw-r----- 1 pordyna pordyna 121G Jun  3 02:17 data.158
4662529094 122G -rw-r----- 1 pordyna pordyna 122G Jun  3 02:17 data.159
4658257141 239G -rw-r----- 1 pordyna pordyna 239G Jun  3 02:17 data.16
4658257164 121G -rw-r----- 1 pordyna pordyna 121G Jun  3 02:17 data.160
4658520128 118G -rw-r----- 1 pordyna pordyna 118G Jun  3 02:17 data.161
4658520111 118G -rw-r----- 1 pordyna pordyna 118G Jun  3 02:17 data.162
4658257228 121G -rw-r----- 1 pordyna pordyna 121G Jun  3 02:17 data.163
4664941038 122G -rw-r----- 1 pordyna pordyna 122G Jun  3 02:17 data.164
4664941051 121G -rw-r----- 1 pordyna pordyna 121G Jun  3 02:17 data.165
4664941081 118G -rw-r----- 1 pordyna pordyna 118G Jun  3 02:17 data.166
4664941039 107G -rw-r----- 1 pordyna pordyna 107G Jun  3 02:17 data.167
4662416781 106G -rw-r----- 1 pordyna pordyna 106G Jun  3 02:17 data.168
4662417128 117G -rw-r----- 1 pordyna pordyna 117G Jun  3 02:17 data.169
4658520525 240G -rw-r----- 1 pordyna pordyna 240G Jun  3 02:17 data.17
4662417127 118G -rw-r----- 1 pordyna pordyna 118G Jun  3 02:17 data.170
4662417129 122G -rw-r----- 1 pordyna pordyna 122G Jun  3 02:17 data.171
4661812922 125G -rw-r----- 1 pordyna pordyna 125G Jun  3 02:17 data.172
4661812847 127G -rw-r----- 1 pordyna pordyna 127G Jun  3 02:17 data.173
4661812774 127G -rw-r----- 1 pordyna pordyna 127G Jun  3 02:17 data.174
4661286737 125G -rw-r----- 1 pordyna pordyna 125G Jun  3 02:17 data.175
2669369497 121G -rw-r----- 1 pordyna pordyna 121G Jun  3 02:17 data.176
2669369499 119G -rw-r----- 1 pordyna pordyna 119G Jun  3 02:17 data.177
2669369496 117G -rw-r----- 1 pordyna pordyna 117G Jun  3 02:17 data.178
2669369495 106G -rw-r----- 1 pordyna pordyna 106G Jun  3 02:17 data.179
4658520486 240G -rw-r----- 1 pordyna pordyna 240G Jun  3 02:17 data.18
4662769535 105G -rw-r----- 1 pordyna pordyna 105G Jun  3 02:17 data.180
4662769534 116G -rw-r----- 1 pordyna pordyna 116G Jun  3 02:17 data.181
4662769536 118G -rw-r----- 1 pordyna pordyna 118G Jun  3 02:17 data.182
4662769532 119G -rw-r----- 1 pordyna pordyna 119G Jun  3 02:17 data.183
4663189792 119G -rw-r----- 1 pordyna pordyna 119G Jun  3 02:17 data.184
4663189818 120G -rw-r----- 1 pordyna pordyna 120G Jun  3 02:17 data.185
4663189800 120G -rw-r----- 1 pordyna pordyna 120G Jun  3 02:17 data.186
4662926925 119G -rw-r----- 1 pordyna pordyna 119G Jun  3 02:17 data.187
4665049743 119G -rw-r----- 1 pordyna pordyna 119G Jun  3 02:17 data.188
4659042629 118G -rw-r----- 1 pordyna pordyna 118G Jun  3 02:17 data.189
4658520495 238G -rw-r----- 1 pordyna pordyna 238G Jun  3 02:17 data.19
4665049742 116G -rw-r----- 1 pordyna pordyna 116G Jun  3 02:17 data.190
4665049744 105G -rw-r----- 1 pordyna pordyna 105G Jun  3 02:17 data.191
4663157431 102G -rw-r----- 1 pordyna pordyna 102G Jun  3 02:17 data.192
4663157429 114G -rw-r----- 1 pordyna pordyna 114G Jun  3 02:17 data.193
4663157432 116G -rw-r----- 1 pordyna pordyna 116G Jun  3 02:17 data.194
4663157430 116G -rw-r----- 1 pordyna pordyna 116G Jun  3 02:17 data.195
4658257898 115G -rw-r----- 1 pordyna pordyna 115G Jun  3 02:17 data.196
4658520255 115G -rw-r----- 1 pordyna pordyna 115G Jun  3 02:17 data.197
4658520279 115G -rw-r----- 1 pordyna pordyna 115G Jun  3 02:17 data.198
4658520293 115G -rw-r----- 1 pordyna pordyna 115G Jun  3 02:17 data.199
4663451216 887G -rw-r----- 1 pordyna pordyna 887G Jun  3 02:17 data.2
4658257386 232G -rw-r----- 1 pordyna pordyna 232G Jun  3 02:17 data.20
4662478819 116G -rw-r----- 1 pordyna pordyna 116G Jun  3 02:17 data.200
4662740996 116G -rw-r----- 1 pordyna pordyna 116G Jun  3 02:17 data.201
4662740999 114G -rw-r----- 1 pordyna pordyna 114G Jun  3 02:17 data.202
4662740997 103G -rw-r----- 1 pordyna pordyna 103G Jun  3 02:17 data.203
3957491356 227G -rw-r----- 1 pordyna pordyna 227G Jun  3 02:17 data.204
3957491374 244G -rw-r----- 1 pordyna pordyna 244G Jun  3 02:17 data.205
3957491376 247G -rw-r----- 1 pordyna pordyna 247G Jun  3 02:17 data.206
3957491375 249G -rw-r----- 1 pordyna pordyna 249G Jun  3 02:17 data.207
4662809161 253G -rw-r----- 1 pordyna pordyna 253G Jun  3 02:17 data.208
4662809162 258G -rw-r----- 1 pordyna pordyna 258G Jun  3 02:17 data.209
4658520980 225G -rw-r----- 1 pordyna pordyna 225G Jun  3 02:17 data.21
4662809163 258G -rw-r----- 1 pordyna pordyna 258G Jun  3 02:17 data.210
4662809165 254G -rw-r----- 1 pordyna pordyna 254G Jun  3 02:17 data.211
4650052886 250G -rw-r----- 1 pordyna pordyna 250G Jun  3 02:17 data.212
4650052907 248G -rw-r----- 1 pordyna pordyna 248G Jun  3 02:17 data.213
4650052796 244G -rw-r----- 1 pordyna pordyna 244G Jun  3 02:17 data.214
4650052903 227G -rw-r----- 1 pordyna pordyna 227G Jun  3 02:17 data.215
4665049745 221G -rw-r----- 1 pordyna pordyna 221G Jun  3 02:17 data.216
4665049747 230G -rw-r----- 1 pordyna pordyna 230G Jun  3 02:17 data.217
4665049746 230G -rw-r----- 1 pordyna pordyna 230G Jun  3 02:17 data.218
4665049748 230G -rw-r----- 1 pordyna pordyna 230G Jun  3 02:17 data.219
4658257562 217G -rw-r----- 1 pordyna pordyna 217G Jun  3 02:17 data.22
4664536762 229G -rw-r----- 1 pordyna pordyna 229G Jun  3 02:17 data.220
4664536765 230G -rw-r----- 1 pordyna pordyna 230G Jun  3 02:17 data.221
4664536763 230G -rw-r----- 1 pordyna pordyna 230G Jun  3 02:17 data.222
4664536764 229G -rw-r----- 1 pordyna pordyna 229G Jun  3 02:17 data.223
4303599953 229G -rw-r----- 1 pordyna pordyna 229G Jun  3 02:17 data.224
4655002848 230G -rw-r----- 1 pordyna pordyna 230G Jun  3 02:17 data.225
4303599868 230G -rw-r----- 1 pordyna pordyna 230G Jun  3 02:17 data.226
 212837076 221G -rw-r----- 1 pordyna pordyna 221G Jun  3 02:17 data.227
4661549414 106G -rw-r----- 1 pordyna pordyna 106G Jun  3 02:17 data.228
4661286814 118G -rw-r----- 1 pordyna pordyna 118G Jun  3 02:17 data.229
4658521013 203G -rw-r----- 1 pordyna pordyna 203G Jun  3 02:17 data.23
4661286853 121G -rw-r----- 1 pordyna pordyna 121G Jun  3 02:17 data.230
4661549399 122G -rw-r----- 1 pordyna pordyna 122G Jun  3 02:17 data.231
4663451228 120G -rw-r----- 1 pordyna pordyna 120G Jun  3 02:17 data.232
4663451227 119G -rw-r----- 1 pordyna pordyna 119G Jun  3 02:17 data.233
4663451229 118G -rw-r----- 1 pordyna pordyna 118G Jun  3 02:17 data.234
4663451230 121G -rw-r----- 1 pordyna pordyna 121G Jun  3 02:17 data.235
4659180565 122G -rw-r----- 1 pordyna pordyna 122G Jun  3 02:17 data.236
4661812546 121G -rw-r----- 1 pordyna pordyna 121G Jun  3 02:17 data.237
4661812518 118G -rw-r----- 1 pordyna pordyna 118G Jun  3 02:17 data.238
4661812590 107G -rw-r----- 1 pordyna pordyna 107G Jun  3 02:17 data.239
4663352824 213G -rw-r----- 1 pordyna pordyna 213G Jun  3 02:17 data.24
4658520809 106G -rw-r----- 1 pordyna pordyna 106G Jun  3 02:17 data.240
4658520968 117G -rw-r----- 1 pordyna pordyna 117G Jun  3 02:17 data.241
4658520770 118G -rw-r----- 1 pordyna pordyna 118G Jun  3 02:17 data.242
4658520967 121G -rw-r----- 1 pordyna pordyna 121G Jun  3 02:17 data.243
3957491391 125G -rw-r----- 1 pordyna pordyna 125G Jun  3 02:17 data.244
3957491395 128G -rw-r----- 1 pordyna pordyna 128G Jun  3 02:17 data.245
3957491393 127G -rw-r----- 1 pordyna pordyna 127G Jun  3 02:17 data.246
3957491385 125G -rw-r----- 1 pordyna pordyna 125G Jun  3 02:17 data.247
4658520730 121G -rw-r----- 1 pordyna pordyna 121G Jun  3 02:17 data.248
4658520733 119G -rw-r----- 1 pordyna pordyna 119G Jun  3 02:17 data.249
4663352807 227G -rw-r----- 1 pordyna pordyna 227G Jun  3 02:17 data.25
4658520734 117G -rw-r----- 1 pordyna pordyna 117G Jun  3 02:17 data.250
4658520741 106G -rw-r----- 1 pordyna pordyna 106G Jun  3 02:17 data.251
4663176486 105G -rw-r----- 1 pordyna pordyna 105G Jun  3 02:17 data.252
4663176803 116G -rw-r----- 1 pordyna pordyna 116G Jun  3 02:17 data.253
4663176801 118G -rw-r----- 1 pordyna pordyna 118G Jun  3 02:17 data.254
4663176805 119G -rw-r----- 1 pordyna pordyna 119G Jun  3 02:17 data.255
4665049749 119G -rw-r----- 1 pordyna pordyna 119G Jun  3 02:17 data.256
4662826430 120G -rw-r----- 1 pordyna pordyna 120G Jun  3 02:17 data.257
4663089814 120G -rw-r----- 1 pordyna pordyna 120G Jun  3 02:17 data.258
3957491402 119G -rw-r----- 1 pordyna pordyna 119G Jun  3 02:17 data.259
4663089958 233G -rw-r----- 1 pordyna pordyna 233G Jun  3 02:17 data.26
4662846142 119G -rw-r----- 1 pordyna pordyna 119G Jun  3 02:17 data.260
4662846108 118G -rw-r----- 1 pordyna pordyna 118G Jun  3 02:17 data.261
4662846141 116G -rw-r----- 1 pordyna pordyna 116G Jun  3 02:17 data.262
4662846107 105G -rw-r----- 1 pordyna pordyna 105G Jun  3 02:17 data.263
4662658170 102G -rw-r----- 1 pordyna pordyna 102G Jun  3 02:17 data.264
4662658171 114G -rw-r----- 1 pordyna pordyna 114G Jun  3 02:17 data.265
4662658169 116G -rw-r----- 1 pordyna pordyna 116G Jun  3 02:17 data.266
4662658172 116G -rw-r----- 1 pordyna pordyna 116G Jun  3 02:17 data.267
4664536761 115G -rw-r----- 1 pordyna pordyna 115G Jun  3 02:17 data.268
4664536759 115G -rw-r----- 1 pordyna pordyna 115G Jun  3 02:17 data.269
4663352834 241G -rw-r----- 1 pordyna pordyna 241G Jun  3 02:17 data.27
4664536760 115G -rw-r----- 1 pordyna pordyna 115G Jun  3 02:17 data.270
4664536743 115G -rw-r----- 1 pordyna pordyna 115G Jun  3 02:17 data.271
4662526936 116G -rw-r----- 1 pordyna pordyna 116G Jun  3 02:17 data.272
4662526938 116G -rw-r----- 1 pordyna pordyna 116G Jun  3 02:17 data.273
4662526937 114G -rw-r----- 1 pordyna pordyna 114G Jun  3 02:17 data.274
4662526939 103G -rw-r----- 1 pordyna pordyna 103G Jun  3 02:17 data.275
4650052626 226G -rw-r----- 1 pordyna pordyna 226G Jun  3 02:17 data.276
4650052627 243G -rw-r----- 1 pordyna pordyna 243G Jun  3 02:17 data.277
4650052633 247G -rw-r----- 1 pordyna pordyna 247G Jun  3 02:17 data.278
4650052628 249G -rw-r----- 1 pordyna pordyna 249G Jun  3 02:17 data.279
4662853199 248G -rw-r----- 1 pordyna pordyna 248G Jun  3 02:17 data.28
4662592816 253G -rw-r----- 1 pordyna pordyna 253G Jun  3 02:17 data.280
4662592814 258G -rw-r----- 1 pordyna pordyna 258G Jun  3 02:17 data.281
4662592815 257G -rw-r----- 1 pordyna pordyna 257G Jun  3 02:17 data.282
4662592813 253G -rw-r----- 1 pordyna pordyna 253G Jun  3 02:17 data.283
4663121459 250G -rw-r----- 1 pordyna pordyna 250G Jun  3 02:17 data.284
4663648086 247G -rw-r----- 1 pordyna pordyna 247G Jun  3 02:17 data.285
4663648098 243G -rw-r----- 1 pordyna pordyna 243G Jun  3 02:17 data.286
4663648103 227G -rw-r----- 1 pordyna pordyna 227G Jun  3 02:17 data.287
4662514120 209G -rw-r----- 1 pordyna pordyna 209G Jun  3 02:17 data.288
4662514130 222G -rw-r----- 1 pordyna pordyna 222G Jun  3 02:17 data.289
4662590438 253G -rw-r----- 1 pordyna pordyna 253G Jun  3 02:17 data.29
4662514129 225G -rw-r----- 1 pordyna pordyna 225G Jun  3 02:17 data.290
4662514131 225G -rw-r----- 1 pordyna pordyna 225G Jun  3 02:17 data.291
3697278801 224G -rw-r----- 1 pordyna pordyna 224G Jun  3 02:17 data.292
3697278830 225G -rw-r----- 1 pordyna pordyna 225G Jun  3 02:17 data.293
3697278917 224G -rw-r----- 1 pordyna pordyna 224G Jun  3 02:17 data.294
3697278944 224G -rw-r----- 1 pordyna pordyna 224G Jun  3 02:17 data.295
4663972798 224G -rw-r----- 1 pordyna pordyna 224G Jun  3 02:17 data.296
4663972795 225G -rw-r----- 1 pordyna pordyna 225G Jun  3 02:17 data.297
4663972794 222G -rw-r----- 1 pordyna pordyna 222G Jun  3 02:17 data.298
4663972796 209G -rw-r----- 1 pordyna pordyna 209G Jun  3 02:17 data.299
4663451218 905G -rw-r----- 1 pordyna pordyna 905G Jun  3 02:17 data.3
4662853205 253G -rw-r----- 1 pordyna pordyna 253G Jun  3 02:17 data.30
4651953874  87G -rw-r----- 1 pordyna pordyna  87G Jun  3 02:17 data.300
4651953875  92G -rw-r----- 1 pordyna pordyna  92G Jun  3 02:17 data.301
4662522239  92G -rw-r----- 1 pordyna pordyna  92G Jun  3 02:17 data.302
4662522238  91G -rw-r----- 1 pordyna pordyna  91G Jun  3 02:17 data.303
4662820265  89G -rw-r----- 1 pordyna pordyna  89G Jun  3 02:17 data.304
4662820309  88G -rw-r----- 1 pordyna pordyna  88G Jun  3 02:17 data.305
4662820409  88G -rw-r----- 1 pordyna pordyna  88G Jun  3 02:17 data.306
4662820312  89G -rw-r----- 1 pordyna pordyna  89G Jun  3 02:17 data.307
4658520639  91G -rw-r----- 1 pordyna pordyna  91G Jun  3 02:17 data.308
4658520593  92G -rw-r----- 1 pordyna pordyna  92G Jun  3 02:17 data.309
4662590432 248G -rw-r----- 1 pordyna pordyna 248G Jun  3 02:17 data.31
4657994243  92G -rw-r----- 1 pordyna pordyna  92G Jun  3 02:17 data.310
4657994016  87G -rw-r----- 1 pordyna pordyna  87G Jun  3 02:17 data.311
4658257797 105G -rw-r----- 1 pordyna pordyna 105G Jun  3 02:17 data.312
4658520662 116G -rw-r----- 1 pordyna pordyna 116G Jun  3 02:17 data.313
4658520722 117G -rw-r----- 1 pordyna pordyna 117G Jun  3 02:17 data.314
4658257836 120G -rw-r----- 1 pordyna pordyna 120G Jun  3 02:17 data.315
4661812615 120G -rw-r----- 1 pordyna pordyna 120G Jun  3 02:17 data.316
4660760453 121G -rw-r----- 1 pordyna pordyna 121G Jun  3 02:17 data.317
4661286852 120G -rw-r----- 1 pordyna pordyna 120G Jun  3 02:17 data.318
4661286460 121G -rw-r----- 1 pordyna pordyna 121G Jun  3 02:17 data.319
4664008450 240G -rw-r----- 1 pordyna pordyna 240G Jun  3 02:17 data.32
4659042690 120G -rw-r----- 1 pordyna pordyna 120G Jun  3 02:17 data.320
4658520681 118G -rw-r----- 1 pordyna pordyna 118G Jun  3 02:17 data.321
4658520763 116G -rw-r----- 1 pordyna pordyna 116G Jun  3 02:17 data.322
4658520605 105G -rw-r----- 1 pordyna pordyna 105G Jun  3 02:17 data.323
4454821628 104G -rw-r----- 1 pordyna pordyna 104G Jun  3 02:17 data.324
4454821630 116G -rw-r----- 1 pordyna pordyna 116G Jun  3 02:17 data.325
4648427848 118G -rw-r----- 1 pordyna pordyna 118G Jun  3 02:17 data.326
4648165372 119G -rw-r----- 1 pordyna pordyna 119G Jun  3 02:17 data.327
4657876615 118G -rw-r----- 1 pordyna pordyna 118G Jun  3 02:17 data.328
4657876616 118G -rw-r----- 1 pordyna pordyna 118G Jun  3 02:17 data.329
4664008452 233G -rw-r----- 1 pordyna pordyna 233G Jun  3 02:17 data.33
4657612825 118G -rw-r----- 1 pordyna pordyna 118G Jun  3 02:17 data.330
4649981123 118G -rw-r----- 1 pordyna pordyna 118G Jun  3 02:17 data.331
 654043744 119G -rw-r----- 1 pordyna pordyna 119G Jun  3 02:17 data.332
 653780909 118G -rw-r----- 1 pordyna pordyna 118G Jun  3 02:17 data.333
 653517139 116G -rw-r----- 1 pordyna pordyna 116G Jun  3 02:17 data.334
 653780908 104G -rw-r----- 1 pordyna pordyna 104G Jun  3 02:17 data.335
4662529100  86G -rw-r----- 1 pordyna pordyna  86G Jun  3 02:17 data.336
4662529101  93G -rw-r----- 1 pordyna pordyna  93G Jun  3 02:17 data.337
4662529099  92G -rw-r----- 1 pordyna pordyna  92G Jun  3 02:17 data.338
4662529098  91G -rw-r----- 1 pordyna pordyna  91G Jun  3 02:17 data.339
4664008453 227G -rw-r----- 1 pordyna pordyna 227G Jun  3 02:17 data.34
4662075566  90G -rw-r----- 1 pordyna pordyna  90G Jun  3 02:17 data.340
4662075516  90G -rw-r----- 1 pordyna pordyna  90G Jun  3 02:17 data.341
4659443853  90G -rw-r----- 1 pordyna pordyna  90G Jun  3 02:17 data.342
4661813013  90G -rw-r----- 1 pordyna pordyna  90G Jun  3 02:17 data.343
4623086444  91G -rw-r----- 1 pordyna pordyna  91G Jun  3 02:17 data.344
4623086474  93G -rw-r----- 1 pordyna pordyna  93G Jun  3 02:17 data.345
4623086493  93G -rw-r----- 1 pordyna pordyna  93G Jun  3 02:17 data.346
4623086494  86G -rw-r----- 1 pordyna pordyna  86G Jun  3 02:17 data.347
4665793899 216G -rw-r----- 1 pordyna pordyna 216G Jun  3 02:17 data.348
4666583324 233G -rw-r----- 1 pordyna pordyna 233G Jun  3 02:17 data.349
4664008451 213G -rw-r----- 1 pordyna pordyna 213G Jun  3 02:17 data.35
4665794214 238G -rw-r----- 1 pordyna pordyna 238G Jun  3 02:17 data.350
4666583325 240G -rw-r----- 1 pordyna pordyna 240G Jun  3 02:17 data.351
4657731284 244G -rw-r----- 1 pordyna pordyna 244G Jun  3 02:17 data.352
4657994306 248G -rw-r----- 1 pordyna pordyna 248G Jun  3 02:17 data.353
4658257122 248G -rw-r----- 1 pordyna pordyna 248G Jun  3 02:17 data.354
4658257148 244G -rw-r----- 1 pordyna pordyna 244G Jun  3 02:17 data.355
4665616735 240G -rw-r----- 1 pordyna pordyna 240G Jun  3 02:17 data.356
4665616738 238G -rw-r----- 1 pordyna pordyna 238G Jun  3 02:17 data.357
4665616737 233G -rw-r----- 1 pordyna pordyna 233G Jun  3 02:17 data.358
4665616736 217G -rw-r----- 1 pordyna pordyna 217G Jun  3 02:17 data.359
4662686989 212G -rw-r----- 1 pordyna pordyna 212G Jun  3 02:17 data.36
4663168715 815G -rw-r----- 1 pordyna pordyna 815G Jun  3 02:17 data.360
4663168718 862G -rw-r----- 1 pordyna pordyna 862G Jun  3 02:17 data.361
4663168716 888G -rw-r----- 1 pordyna pordyna 888G Jun  3 02:17 data.362
4663168717 906G -rw-r----- 1 pordyna pordyna 906G Jun  3 02:17 data.363
4663513634 913G -rw-r----- 1 pordyna pordyna 913G Jun  3 02:17 data.364
4663513635 911G -rw-r----- 1 pordyna pordyna 911G Jun  3 02:17 data.365
4663513637 911G -rw-r----- 1 pordyna pordyna 911G Jun  3 02:17 data.366
4663513636 911G -rw-r----- 1 pordyna pordyna 911G Jun  3 02:17 data.367
4658520177 903G -rw-r----- 1 pordyna pordyna 903G Jun  3 02:17 data.368
4658520232 886G -rw-r----- 1 pordyna pordyna 886G Jun  3 02:17 data.369
4662424524 227G -rw-r----- 1 pordyna pordyna 227G Jun  3 02:17 data.37
4658520188 862G -rw-r----- 1 pordyna pordyna 862G Jun  3 02:17 data.370
4658520230 817G -rw-r----- 1 pordyna pordyna 817G Jun  3 02:17 data.371
4662592817 203G -rw-r----- 1 pordyna pordyna 203G Jun  3 02:17 data.372
4662592818 218G -rw-r----- 1 pordyna pordyna 218G Jun  3 02:17 data.373
4662592820 225G -rw-r----- 1 pordyna pordyna 225G Jun  3 02:17 data.374
4662592819 233G -rw-r----- 1 pordyna pordyna 233G Jun  3 02:17 data.375
4661630222 239G -rw-r----- 1 pordyna pordyna 239G Jun  3 02:17 data.376
4661630224 240G -rw-r----- 1 pordyna pordyna 240G Jun  3 02:17 data.377
4661630237 240G -rw-r----- 1 pordyna pordyna 240G Jun  3 02:17 data.378
4661630230 239G -rw-r----- 1 pordyna pordyna 239G Jun  3 02:17 data.379
4662424459 234G -rw-r----- 1 pordyna pordyna 234G Jun  3 02:17 data.38
4661438303 233G -rw-r----- 1 pordyna pordyna 233G Jun  3 02:17 data.380
4661438307 225G -rw-r----- 1 pordyna pordyna 225G Jun  3 02:17 data.381
4661438306 217G -rw-r----- 1 pordyna pordyna 217G Jun  3 02:17 data.382
4661438305 203G -rw-r----- 1 pordyna pordyna 203G Jun  3 02:17 data.383
4623086435 213G -rw-r----- 1 pordyna pordyna 213G Jun  3 02:17 data.384
4623086443 227G -rw-r----- 1 pordyna pordyna 227G Jun  3 02:17 data.385
4623086439 233G -rw-r----- 1 pordyna pordyna 233G Jun  3 02:17 data.386
4623086441 241G -rw-r----- 1 pordyna pordyna 241G Jun  3 02:17 data.387
3560482913 248G -rw-r----- 1 pordyna pordyna 248G Jun  3 02:17 data.388
3560482916 253G -rw-r----- 1 pordyna pordyna 253G Jun  3 02:17 data.389
4662686986 240G -rw-r----- 1 pordyna pordyna 240G Jun  3 02:17 data.39
3560482918 253G -rw-r----- 1 pordyna pordyna 253G Jun  3 02:17 data.390
3560482919 248G -rw-r----- 1 pordyna pordyna 248G Jun  3 02:17 data.391
4659181457 241G -rw-r----- 1 pordyna pordyna 241G Jun  3 02:17 data.392
4661812513 233G -rw-r----- 1 pordyna pordyna 233G Jun  3 02:17 data.393
4661549720 227G -rw-r----- 1 pordyna pordyna 227G Jun  3 02:17 data.394
4661286581 213G -rw-r----- 1 pordyna pordyna 213G Jun  3 02:17 data.395
4663030005 212G -rw-r----- 1 pordyna pordyna 212G Jun  3 02:17 data.396
4663557072 227G -rw-r----- 1 pordyna pordyna 227G Jun  3 02:17 data.397
4664345712 234G -rw-r----- 1 pordyna pordyna 234G Jun  3 02:17 data.398
4663029916 240G -rw-r----- 1 pordyna pordyna 240G Jun  3 02:17 data.399
4662809169 913G -rw-r----- 1 pordyna pordyna 913G Jun  3 02:17 data.4
4661812977 248G -rw-r----- 1 pordyna pordyna 248G Jun  3 02:17 data.40
4604462786 248G -rw-r----- 1 pordyna pordyna 248G Jun  3 02:17 data.400
4604462783 253G -rw-r----- 1 pordyna pordyna 253G Jun  3 02:17 data.401
4604462784 253G -rw-r----- 1 pordyna pordyna 253G Jun  3 02:17 data.402
4604462782 248G -rw-r----- 1 pordyna pordyna 248G Jun  3 02:17 data.403
4662649754 241G -rw-r----- 1 pordyna pordyna 241G Jun  3 02:17 data.404
4662649755 234G -rw-r----- 1 pordyna pordyna 234G Jun  3 02:17 data.405
4662649756 227G -rw-r----- 1 pordyna pordyna 227G Jun  3 02:17 data.406
4662649757 212G -rw-r----- 1 pordyna pordyna 212G Jun  3 02:17 data.407
4662521004 202G -rw-r----- 1 pordyna pordyna 202G Jun  3 02:17 data.408
4662521007 216G -rw-r----- 1 pordyna pordyna 216G Jun  3 02:17 data.409
4661812969 253G -rw-r----- 1 pordyna pordyna 253G Jun  3 02:17 data.41
4662521005 224G -rw-r----- 1 pordyna pordyna 224G Jun  3 02:17 data.410
4662520935 231G -rw-r----- 1 pordyna pordyna 231G Jun  3 02:17 data.411
4662075487 238G -rw-r----- 1 pordyna pordyna 238G Jun  3 02:17 data.412
4660759957 244G -rw-r----- 1 pordyna pordyna 244G Jun  3 02:17 data.413
4661549776 245G -rw-r----- 1 pordyna pordyna 245G Jun  3 02:17 data.414
4661286821 238G -rw-r----- 1 pordyna pordyna 238G Jun  3 02:17 data.415
4645099411 231G -rw-r----- 1 pordyna pordyna 231G Jun  3 02:17 data.416
4658520674 224G -rw-r----- 1 pordyna pordyna 224G Jun  3 02:17 data.417
4658520975 217G -rw-r----- 1 pordyna pordyna 217G Jun  3 02:17 data.418
4657994487 203G -rw-r----- 1 pordyna pordyna 203G Jun  3 02:17 data.419
4661813155 253G -rw-r----- 1 pordyna pordyna 253G Jun  3 02:17 data.42
3697541129 831G -rw-r----- 1 pordyna pordyna 831G Jun  3 02:17 data.420
3697278969 871G -rw-r----- 1 pordyna pordyna 871G Jun  3 02:17 data.421
3697541136 886G -rw-r----- 1 pordyna pordyna 886G Jun  3 02:17 data.422
3697278948 901G -rw-r----- 1 pordyna pordyna 901G Jun  3 02:17 data.423
4663189834 926G -rw-r----- 1 pordyna pordyna 926G Jun  3 02:17 data.424
4664768830 963G -rw-r----- 1 pordyna pordyna 963G Jun  3 02:17 data.425
4663189829 964G -rw-r----- 1 pordyna pordyna 964G Jun  3 02:17 data.426
4664768829 927G -rw-r----- 1 pordyna pordyna 927G Jun  3 02:17 data.427
1921385243 902G -rw-r----- 1 pordyna pordyna 902G Jun  3 02:17 data.428
1921385295 887G -rw-r----- 1 pordyna pordyna 887G Jun  3 02:17 data.429
4661813125 248G -rw-r----- 1 pordyna pordyna 248G Jun  3 02:17 data.43
1921385213 872G -rw-r----- 1 pordyna pordyna 872G Jun  3 02:17 data.430
1921385296 834G -rw-r----- 1 pordyna pordyna 834G Jun  3 02:17 data.431
4663451223 240G -rw-r----- 1 pordyna pordyna 240G Jun  3 02:17 data.44
4663451224 233G -rw-r----- 1 pordyna pordyna 233G Jun  3 02:17 data.45
4663451225 226G -rw-r----- 1 pordyna pordyna 226G Jun  3 02:17 data.46
4663451226 212G -rw-r----- 1 pordyna pordyna 212G Jun  3 02:17 data.47
 895495621 202G -rw-r----- 1 pordyna pordyna 202G Jun  3 02:17 data.48
4662707653 216G -rw-r----- 1 pordyna pordyna 216G Jun  3 02:17 data.49
4662809172 912G -rw-r----- 1 pordyna pordyna 912G Jun  3 02:17 data.5
4662707633 224G -rw-r----- 1 pordyna pordyna 224G Jun  3 02:17 data.50
4662707611 230G -rw-r----- 1 pordyna pordyna 230G Jun  3 02:17 data.51
2037671215 238G -rw-r----- 1 pordyna pordyna 238G Jun  3 02:17 data.52
2037671217 244G -rw-r----- 1 pordyna pordyna 244G Jun  3 02:17 data.53
2037671218 245G -rw-r----- 1 pordyna pordyna 245G Jun  3 02:17 data.54
2037671216 238G -rw-r----- 1 pordyna pordyna 238G Jun  3 02:17 data.55
4662589384 230G -rw-r----- 1 pordyna pordyna 230G Jun  3 02:17 data.56
4662589403 223G -rw-r----- 1 pordyna pordyna 223G Jun  3 02:17 data.57
4662589386 216G -rw-r----- 1 pordyna pordyna 216G Jun  3 02:17 data.58
4662589429 202G -rw-r----- 1 pordyna pordyna 202G Jun  3 02:17 data.59
4662809171 911G -rw-r----- 1 pordyna pordyna 911G Jun  3 02:17 data.6
4666583326 833G -rw-r----- 1 pordyna pordyna 833G Jun  3 02:17 data.60
4666583328 872G -rw-r----- 1 pordyna pordyna 872G Jun  3 02:17 data.61
4666583284 886G -rw-r----- 1 pordyna pordyna 886G Jun  3 02:17 data.62
4666583327 900G -rw-r----- 1 pordyna pordyna 900G Jun  3 02:17 data.63
3082902684 924G -rw-r----- 1 pordyna pordyna 924G Jun  3 02:17 data.64
3082902878 962G -rw-r----- 1 pordyna pordyna 962G Jun  3 02:17 data.65
3082902894 963G -rw-r----- 1 pordyna pordyna 963G Jun  3 02:17 data.66
3082903309 928G -rw-r----- 1 pordyna pordyna 928G Jun  3 02:17 data.67
4658520314 903G -rw-r----- 1 pordyna pordyna 903G Jun  3 02:17 data.68
4658520363 888G -rw-r----- 1 pordyna pordyna 888G Jun  3 02:17 data.69
4662809167 911G -rw-r----- 1 pordyna pordyna 911G Jun  3 02:17 data.7
4658520342 872G -rw-r----- 1 pordyna pordyna 872G Jun  3 02:17 data.70
4658520390 833G -rw-r----- 1 pordyna pordyna 833G Jun  3 02:17 data.71
4664536768 208G -rw-r----- 1 pordyna pordyna 208G Jun  3 02:17 data.72
4664536769 222G -rw-r----- 1 pordyna pordyna 222G Jun  3 02:17 data.73
4664536767 225G -rw-r----- 1 pordyna pordyna 225G Jun  3 02:17 data.74
4664536766 225G -rw-r----- 1 pordyna pordyna 225G Jun  3 02:17 data.75
4658257739 224G -rw-r----- 1 pordyna pordyna 224G Jun  3 02:17 data.76
4658520243 224G -rw-r----- 1 pordyna pordyna 224G Jun  3 02:17 data.77
4658257749 225G -rw-r----- 1 pordyna pordyna 225G Jun  3 02:17 data.78
4658520236 223G -rw-r----- 1 pordyna pordyna 223G Jun  3 02:17 data.79
4662707702 905G -rw-r----- 1 pordyna pordyna 905G Jun  3 02:17 data.8
4662999579 225G -rw-r----- 1 pordyna pordyna 225G Jun  3 02:17 data.80
4662999578 226G -rw-r----- 1 pordyna pordyna 226G Jun  3 02:17 data.81
4662999544 222G -rw-r----- 1 pordyna pordyna 222G Jun  3 02:17 data.82
4662999580 209G -rw-r----- 1 pordyna pordyna 209G Jun  3 02:17 data.83
4661813216  87G -rw-r----- 1 pordyna pordyna  87G Jun  3 02:17 data.84
4662075486  92G -rw-r----- 1 pordyna pordyna  92G Jun  3 02:17 data.85
4662075466  92G -rw-r----- 1 pordyna pordyna  92G Jun  3 02:17 data.86
4662075470  91G -rw-r----- 1 pordyna pordyna  91G Jun  3 02:17 data.87
4659970821  89G -rw-r----- 1 pordyna pordyna  89G Jun  3 02:17 data.88
4661023237  88G -rw-r----- 1 pordyna pordyna  88G Jun  3 02:17 data.89
4662707684 887G -rw-r----- 1 pordyna pordyna 887G Jun  3 02:17 data.9
4661023222  88G -rw-r----- 1 pordyna pordyna  88G Jun  3 02:17 data.90
4661286618  89G -rw-r----- 1 pordyna pordyna  89G Jun  3 02:17 data.91
4662569258  91G -rw-r----- 1 pordyna pordyna  91G Jun  3 02:17 data.92
4662569256  92G -rw-r----- 1 pordyna pordyna  92G Jun  3 02:17 data.93
4662569253  92G -rw-r----- 1 pordyna pordyna  92G Jun  3 02:17 data.94
4662569254  87G -rw-r----- 1 pordyna pordyna  87G Jun  3 02:17 data.95
4662826425 105G -rw-r----- 1 pordyna pordyna 105G Jun  3 02:17 data.96
4662826428 116G -rw-r----- 1 pordyna pordyna 116G Jun  3 02:17 data.97
4662826426 118G -rw-r----- 1 pordyna pordyna 118G Jun  3 02:17 data.98
4662563329 120G -rw-r----- 1 pordyna pordyna 120G Jun  3 02:17 data.99
4663451220 575G -rw-r----- 1 pordyna pordyna 575G Jun  3 02:17 md.0
4663451221 178M -rw-r----- 1 pordyna pordyna 178M Jun  3 02:24 md.idx
4663451219  13G -rw-r----- 1 pordyna pordyna  13G Jun  3 02:17 mmd.0
4662811599 224K -rw-r----- 1 pordyna pordyna 217K Jun  3 02:24 profiling.json

@franzpoeschel
Copy link
Contributor

Half a terabyte of metadata, that's probably a new record.. :D This clearly explains the bad runtime behavior that you observe.
Do you have an estimation just how many iterations there are in this file? It might be interesting to see the file sizes for a run with a low number of output steps, maybe 10 or so, and then see how the scaling goes.

One solution to that might be to use variable-based encoding and see how that behaves, but it's not yet available on our side.
Creating a new set of variables and attributes per step is not really the intended way for using ADIOS2, but half a terabyte still seems excessive. I'm not sure if we might be looking at an ADIOS2 bug here..
The best workaround for the moment is probably to stick with file-based encoding (note that BP5 has a far lower overhead per file than BP4 had).

(Side note: There is also one workflow issue with variable/group-based encoding that is not yet really solved yet, which is checkpoint-restart workflows. When restarting from a checkpoint, some outputs might be written again. When appending to a group/variable-based Series, this will lead to duplication of steps. openPMD will currently skip any duplicate iterations and you will see only the first instance. Also I just noticed that in PIConGPU, the Append mode still needs to be added. Currently when restarting from a checkpoint in PIConGPU, a group-encoded dataset will be entirely overwritten. I'll add a fix in PIConGPU.)

@pordyna
Copy link
Contributor Author

pordyna commented Jun 7, 2023

I think there should be sth around 2400 iterations in there... In that case, I will just stick to file based encoding for now. Do you still want me to check anything else?

@franzpoeschel
Copy link
Contributor

That is still ~250MB of metadata per iteration. If you continue with file-based encoding, can you check the metadata size there?

@pordyna
Copy link
Contributor Author

pordyna commented Jun 7, 2023

Sure, I will do

@pordyna
Copy link
Contributor Author

pordyna commented Jun 7, 2023

@franzpoeschel I did also some memory profiling on reading iterations with READ_LINEAR. The metadata reading overhead is not that big but still quite large.
This time with the following test script:

from pathlib import Path
import openpmd_api as io
base_path = Path(<setup_path>")
path = base_path / 'simOutput/openPMD/simData.bp5'
series = io.Series(str(path), io.Access.read_linear,  "{\"defer_iteration_parsing\": true}")
for ii, api_iteration in enumerate(series.read_iterations()):
    print(ii, api_iteration.iteration_index, list(api_iteration.meshes))
    if 'B' in list(api_iteration.meshes):
        mr = api_iteration.meshes['B']
        mrc = mr['x']
        data = mrc[60, 60 ,60]
        series.flush()
        print(ii, api_iteration.iteration_index, data)
    api_iteration.close()
    if ii==10:
        break

Output:

0 0 ['B', 'E', 'H_all_density', 'H_all_energyDensity', 'H_all_macroParticleCounter', 'J', 'e_all_density', 'e_all_densityOverGammaSquared', 'e_all_energyDensity', 'e_all_macroParticleCounter', 'picongpu_idProvider']
0 0 0.0
1 1 ['B', 'E', 'H_all_density', 'H_all_energyDensity', 'H_all_macroParticleCounter', 'J', 'e_all_density', 'e_all_densityOverGammaSquared', 'e_all_energyDensity', 'e_all_macroParticleCounter', 'picongpu_idProvider']
1 1 0.0
2 252 ['B', 'E', 'H_all_density', 'H_all_energyDensity', 'H_all_macroParticleCounter', 'J', 'e_all_density', 'e_all_densityOverGammaSquared', 'e_all_energyDensity', 'e_all_macroParticleCounter', 'picongpu_idProvider']
2 252 6.5194583e-09
3 450 ['picongpu_idProvider']
4 504 ['B', 'E', 'H_all_density', 'H_all_energyDensity', 'H_all_macroParticleCounter', 'J', 'e_all_density', 'e_all_densityOverGammaSquared', 'e_all_energyDensity', 'e_all_macroParticleCounter', 'picongpu_idProvider']
4 504 2.7197627e-08
5 756 ['B', 'E', 'H_all_density', 'H_all_energyDensity', 'H_all_macroParticleCounter', 'J', 'e_all_density', 'e_all_densityOverGammaSquared', 'e_all_energyDensity', 'e_all_macroParticleCounter', 'picongpu_idProvider']
5 756 8.3078085e-08
6 900 ['picongpu_idProvider']
7 1008 ['B', 'E', 'H_all_density', 'H_all_energyDensity', 'H_all_macroParticleCounter', 'J', 'e_all_density', 'e_all_densityOverGammaSquared', 'e_all_energyDensity', 'e_all_macroParticleCounter', 'picongpu_idProvider']
7 1008 1.896294e-07
8 1260 ['B', 'E', 'H_all_density', 'H_all_energyDensity', 'H_all_macroParticleCounter', 'J', 'e_all_density', 'e_all_densityOverGammaSquared', 'e_all_energyDensity', 'e_all_macroParticleCounter', 'picongpu_idProvider']
8 1260 3.5043013e-07
9 1350 ['picongpu_idProvider']
10 1512 ['B', 'E', 'H_all_density', 'H_all_energyDensity', 'H_all_macroParticleCounter', 'J', 'e_all_density', 'e_all_densityOverGammaSquared', 'e_all_energyDensity', 'e_all_macroParticleCounter', 'picongpu_idProvider']
10 1512 5.9877743e-07

memory profile with mprof :
mprof_read_liner

Also did run with memray run --native, not till the end since there are a lot more single allocations now and the program needs more than 1h to do series.read_iterations() ^^.
memory usage during memray run:
newplot(1)
interesting part of the summary:
image

Full Stack in flamegraph:
memray-flamegraph-test_open_series_lin.py.147467.html.txt

@franzpoeschel
Copy link
Contributor

To be fair, given the size of the metadata file this is very likely not a problem on the read side. The behavior shown by your profiles is to be expected. I also expect a memory profile of bpls -t to show similar results.

What I'm wondering about: In the first profile, I think that the peaks are the single iterations? That profile looks "good" at least insofar as that the memory use is constant. In the second profile however, the memory use seems to be growing over time, which should not be, except if the profile is cumulative..?

Also I just noticed that in PIConGPU, the Append mode still needs to be added.

.. it seems that there is a bug in ADIOS2 making the Append unusable currently in combination with a memory-saving API that PIConGPU uses ornladios/ADIOS2#3656
It can be deactivated at the price of increased host memory use, but this means that the Append mode can currently at most be added as an opt-in.

@pordyna
Copy link
Contributor Author

pordyna commented Jun 7, 2023

The second profile ends somewhere in the second ramp before reaching the spikes in the first one. I killed it before it finished creating the iterator (still in the series.read_iterations() call in for ii, api_iteration in enumerate(series.read_iterations()):, before the loop starts). It is just that this second ramp are many small allocations and memray is tracking all of them, this makes it extremely slow. mprof is only sampling the total process memory consumption, so it's almost no overhead.

@pordyna
Copy link
Contributor Author

pordyna commented Jun 7, 2023

Well restarting from checkpoints is quite usefull^^, so I guess file based encoding is the way to go for now.

@pordyna pordyna mentioned this issue Jun 8, 2023
Closed
@pordyna
Copy link
Contributor Author

pordyna commented Jun 30, 2023

@franzpoeschel just to update on this, I had a simulation run with file-based encoding and the metadata size is slightly below 8 MB per iteration. The output was now slightly different and split into 3 series (3 plugin instances), but the 8 MB are the same for all 3 series.

@franzpoeschel
Copy link
Contributor

I've tested this out in a PIConGPU run with 512 instances, writing data 50 times. There really seems to be a noticeable difference in metadata buildup in the BP5 engine depending on encoding.

Using variable-based encoding with #1310 (don't use variable-based encoding before that PR is merged, its current implementation will be removed and datasets will be unreadable), the file md.0 has size 198 MB and accessing the dataset via bpls goes reasonably fast. The encoding produces a dataset like this:

> bpls simData.bp5/   
  float     /data/fields/B/x                                      51*{96, 768, 768}
  float     /data/fields/B/y                                      51*{96, 768, 768}
  float     /data/fields/B/z                                      51*{96, 768, 768}
  float     /data/fields/E/x                                      51*{96, 768, 768}
  float     /data/fields/E/y                                      51*{96, 768, 768}
  float     /data/fields/E/z                                      51*{96, 768, 768}
  float     /data/fields/e_all_chargeDensity                      51*{96, 768, 768}
  float     /data/fields/e_all_energyDensity                      51*{96, 768, 768}
  float     /data/fields/i_all_chargeDensity                      51*{96, 768, 768}
  float     /data/fields/i_all_energyDensity                      51*{96, 768, 768}
  uint64_t  /data/fields/picongpu_idProvider/nextId               51*{8, 8, 8}
  uint64_t  /data/fields/picongpu_idProvider/startId              51*{8, 8, 8}
  float     /data/particles/e/momentum/x                          51*{1415577600}
  float     /data/particles/e/momentum/y                          51*{1415577600}
  float     /data/particles/e/momentum/z                          51*{1415577600}
  uint64_t  /data/particles/e/particlePatches/extent/x            51*{512}
  uint64_t  /data/particles/e/particlePatches/extent/y            51*{512}
  uint64_t  /data/particles/e/particlePatches/extent/z            51*{512}
  uint64_t  /data/particles/e/particlePatches/numParticles        51*{512}
  uint64_t  /data/particles/e/particlePatches/numParticlesOffset  51*{512}
  uint64_t  /data/particles/e/particlePatches/offset/x            51*{512}
  uint64_t  /data/particles/e/particlePatches/offset/y            51*{512}
  uint64_t  /data/particles/e/particlePatches/offset/z            51*{512}
  float     /data/particles/e/position/x                          51*{1415577600}
  float     /data/particles/e/position/y                          51*{1415577600}
  float     /data/particles/e/position/z                          51*{1415577600}
  int32_t   /data/particles/e/positionOffset/x                    51*{1415577600}
  int32_t   /data/particles/e/positionOffset/y                    51*{1415577600}
  int32_t   /data/particles/e/positionOffset/z                    51*{1415577600}
  float     /data/particles/e/weighting                           51*{1415577600}
  float     /data/particles/i/momentum/x                          51*{1415577600}
  float     /data/particles/i/momentum/y                          51*{1415577600}
  float     /data/particles/i/momentum/z                          51*{1415577600}
  uint64_t  /data/particles/i/particlePatches/extent/x            51*{512}
  uint64_t  /data/particles/i/particlePatches/extent/y            51*{512}
  uint64_t  /data/particles/i/particlePatches/extent/z            51*{512}
  uint64_t  /data/particles/i/particlePatches/numParticles        51*{512}
  uint64_t  /data/particles/i/particlePatches/numParticlesOffset  51*{512}
  uint64_t  /data/particles/i/particlePatches/offset/x            51*{512}
  uint64_t  /data/particles/i/particlePatches/offset/y            51*{512}
  uint64_t  /data/particles/i/particlePatches/offset/z            51*{512}
  float     /data/particles/i/position/x                          51*{1415577600}
  float     /data/particles/i/position/y                          51*{1415577600}
  float     /data/particles/i/position/z                          51*{1415577600}
  int32_t   /data/particles/i/positionOffset/x                    51*{1415577600}
  int32_t   /data/particles/i/positionOffset/y                    51*{1415577600}
  int32_t   /data/particles/i/positionOffset/z                    51*{1415577600}
  float     /data/particles/i/weighting                           51*{1415577600}

So, instead of creating new variables and attributes for each step, the old ones are reused. I guess that this is what makes the difference compared to the group-based encoding which creates new variables and attributes for each step, has an md.0 file of size 2.3GB and takes noticeable time for reading with bpls:

  float     /data/0/fields/B/x                                         {96, 768, 768}
  float     /data/0/fields/B/y                                         {96, 768, 768}
  float     /data/0/fields/B/z                                         {96, 768, 768}
  float     /data/0/fields/E/x                                         {96, 768, 768}
  float     /data/0/fields/E/y                                         {96, 768, 768}
  float     /data/0/fields/E/z                                         {96, 768, 768}
  float     /data/0/fields/e_all_chargeDensity                         {96, 768, 768}
  float     /data/0/fields/e_all_energyDensity                         {96, 768, 768}
  float     /data/0/fields/i_all_chargeDensity                         {96, 768, 768}
  float     /data/0/fields/i_all_energyDensity                         {96, 768, 768}
  uint64_t  /data/0/fields/picongpu_idProvider/nextId                  {8, 8, 8}
  uint64_t  /data/0/fields/picongpu_idProvider/startId                 {8, 8, 8}
  float     /data/0/particles/e/momentum/x                             {1415577600}
  float     /data/0/particles/e/momentum/y                             {1415577600}
  float     /data/0/particles/e/momentum/z                             {1415577600}
  uint64_t  /data/0/particles/e/particlePatches/extent/x               {512}
  uint64_t  /data/0/particles/e/particlePatches/extent/y               {512}
  uint64_t  /data/0/particles/e/particlePatches/extent/z               {512}
  uint64_t  /data/0/particles/e/particlePatches/numParticles           {512}
  uint64_t  /data/0/particles/e/particlePatches/numParticlesOffset     {512}
  uint64_t  /data/0/particles/e/particlePatches/offset/x               {512}
  uint64_t  /data/0/particles/e/particlePatches/offset/y               {512}
  uint64_t  /data/0/particles/e/particlePatches/offset/z               {512}
  float     /data/0/particles/e/position/x                             {1415577600}
  float     /data/0/particles/e/position/y                             {1415577600}
  float     /data/0/particles/e/position/z                             {1415577600}
  int32_t   /data/0/particles/e/positionOffset/x                       {1415577600}
  int32_t   /data/0/particles/e/positionOffset/y                       {1415577600}
  int32_t   /data/0/particles/e/positionOffset/z                       {1415577600}
  float     /data/0/particles/e/weighting                              {1415577600}
  float     /data/0/particles/i/momentum/x                             {1415577600}
  float     /data/0/particles/i/momentum/y                             {1415577600}
  float     /data/0/particles/i/momentum/z                             {1415577600}
  uint64_t  /data/0/particles/i/particlePatches/extent/x               {512}
  uint64_t  /data/0/particles/i/particlePatches/extent/y               {512}
  uint64_t  /data/0/particles/i/particlePatches/extent/z               {512}
  uint64_t  /data/0/particles/i/particlePatches/numParticles           {512}
  uint64_t  /data/0/particles/i/particlePatches/numParticlesOffset     {512}
  uint64_t  /data/0/particles/i/particlePatches/offset/x               {512}
  uint64_t  /data/0/particles/i/particlePatches/offset/y               {512}
  uint64_t  /data/0/particles/i/particlePatches/offset/z               {512}
  float     /data/0/particles/i/position/x                             {1415577600}
  float     /data/0/particles/i/position/y                             {1415577600}
  float     /data/0/particles/i/position/z                             {1415577600}
  int32_t   /data/0/particles/i/positionOffset/x                       {1415577600}
  int32_t   /data/0/particles/i/positionOffset/y                       {1415577600}
  int32_t   /data/0/particles/i/positionOffset/z                       {1415577600}
  float     /data/0/particles/i/weighting                              {1415577600}
  float     /data/100/fields/B/x                                       {__, __, __}
  float     /data/100/fields/B/y                                       {__, __, __}
  float     /data/100/fields/B/z                                       {__, __, __}
  ...

This kind of layout has never been really the encouraged data encoding in ADIOS2 which is why we are adding the variable-based encoding.

The file-based run is still in the queue, but I imagine that the size of md.0 in a single iteration will be something around 50MB, i.e. 2.3GB/50.

@franzpoeschel
Copy link
Contributor

For the file-based encoding, md.0 has size 13MB, so it seems like the metadata size actually grows overproportionally.

For the moment, this is confirmation that the group-based encoding is not ideal for ADIOS2, however I will bring this up in the next time I talk to the devs.

@pordyna
Copy link
Contributor Author

pordyna commented Jul 6, 2023

@franzpoeschel thank you for looking into that. Should we maybe add a warning/note box to the openPMD plugin documentation?

@pordyna
Copy link
Contributor Author

pordyna commented Jul 6, 2023

For the file-based encoding, md.0 has size 13MB, so it seems like the metadata size actually grows overproportionally.

This is also what I saw.

@franzpoeschel
Copy link
Contributor

We don't actively promote group-based encoding in the PIConGPU documentation. I will wait until I have talked to the ADIOS2 devs, and adapt the documentation based on that.

@franzpoeschel franzpoeschel mentioned this issue Aug 8, 2023
Merged
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
backend: ADIOS2 bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@franzpoeschel @pordyna