Skip to content

Releases: ornladios/ADIOS2

v2.10.0

03 Apr 22:44
@vicentebolea vicentebolea
v2.10.0
6b7b4e4
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v2.10.0 Latest
Latest

What's new in 2.10?

This is a major release with new features and lots of bug fixes. The main new feature is the new Python API.

Python

Before, ADIOS had two separate APIs for Python. The low-level ("Full") API was written with Pybind11 and directly mimicked the C++ API. The high-level API was another, smaller, and more pythonesque API that allowed for easier scripting with Python. The main problems with these two were that they were independent, and that the high-level API was not complete. Once a developer needed a feature only available in the full API, they had to start from scratch writing a script with the full API.

In 2.10, there is officially one Python API, written in Python, which in turn uses the old Pybind11 classes. The new API combines the high-level features of the old high-level API and the full feature set of the low-level bindings.

Note:

Old scripts that used the full API can still run without almost any modification, just change the import line from import adios2 to import adios2.bindings as adios2

Old scripts that used the high-level API must be modified to make them work with the new API, see Transition from old API to new API: Transition from old API to new API

See Python API

New/updated notable features

  • BP5 is supported on Windows now
  • SST and DataMan staging engines are GPU-Aware now
  • SYCL support added for Intel GPUs (besides CUDA and HIP for NVidia and AMD GPUs)
  • the SST/libfabric data transport now works on Frontier (besides the MPI data transport)

New experimental features

What's Changed

Read more

Contributors

eschnett, vicentebolea, and 14 other contributors
Assets 2

v2.10.0-rc1

30 Dec 01:28
@vicentebolea vicentebolea
v2.10.0-rc1
68f8b41
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
v2.10.0-rc1

What's Changed

Read more

Contributors

eschnett, vicentebolea, and 14 other contributors
Assets 2

ADIOS v2.9.2 Production Release

01 Nov 22:12
@vicentebolea vicentebolea
v2.9.2
83cb06a
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
ADIOS v2.9.2 Production Release

Breaking changes

BP5 is the default engine wince 2.9.0 that behaves differently from BP3/BP4. Read the notes in the 2.9.0 release.

What's Changed

Full Changelog: v2.9.1...v2.9.2

Contributors

vicentebolea, eisenhauer, and 2 other contributors
Assets 2

ADIOS v2.9.1 Production Release

03 Aug 16:34
@vicentebolea vicentebolea
v2.9.1
d143154
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
ADIOS v2.9.1 Production Release

Breaking changes

BP5 is the default engine wince 2.9.0 that behaves differently from BP3/BP4. Read the notes in the 2.9.0 release.

Pull Requests backported

PR Title
#3676 Version constants plus functions to retrieve list of ...
#3697 build(deps): bump cryptography from 41.0.0 to 41.0.2 in /docs
#3698 added gitattributes file
#3704 build(deps): bump pygments from 2.14.0 to 2.15.0 in /docs
#3712 build(deps): bump certifi from 2022.12.7 to 2023.7.22 in /docs
#3687 Fix for the issue #3646.
#3699 Update files with new EOL
#3690 Merge pull request #3689 from vicentebolea/fix-evpath-modules
#3689 cmake,evpath: namespace modules install dir
#3684 Fix memory leak when there are Joined Arrays in streaming mode
#3652 work around with include <unistd.h> on windows
#3650 Bump cryptography from 39.0.1 to 41.0.0 in /docs
#3673 ci,ascent: disable install tests
#3681 Fix: std::min w/ windows.h in C-Blosc2
#3657 Allow Span in files opened for Append
#3623 ci: use nvcc_wrapper in adiosKokkos
#3619 Backports bp5 bugfix
#3605 Adding missing HIP specific functionality
#3607 Backport: Heat Transfer Example: MPI Datatype #3593
#3593 Heat Transfer Example: MPI Datatype
#3591 Implement BP5 (and BP4) reader-side memory selection, do testing

Full Changelog: v2.9.0...v2.9.1

Assets 2

ADIOS v2.9.0 Production Release

30 Mar 20:06
@vicentebolea vicentebolea
v2.9.0
aac4a45
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
ADIOS v2.9.0 Production Release

Summary

This is a major release with new features and lots of bug fixes.

Breaking changes

BP5 is the default engine that behaves differently from BP3/BP4. Read mode is proper streaming mode, i.e., Open does not process the metadata and hence io.AvailableVariables() will return empty. Variables and attributes are only available after engine.BeginStep(), and only those that exist in the given step. Also, because of this, the pointer returned by io.InquireVariable() becomes invalid after EndStep. Note, that there is now a new ReadRandomAccess mode to process all metadata in Open and to see all the variables with all their steps at once.

General

  • GPU-Aware I/O enabled by using Kokkos. Device pointers can be passed to Put()/Get() calls directly. Kokkos 3.7.x required for this release. Works with CUDA, HIP and Kokkos applications. https://adios2.readthedocs.io/en/latest/advanced/gpu_aware.html#gpu-aware-i-o
  • GPU-compression. MGARD and ZFP operators can compress data on GPU if they are built for GPU. MGARD operator can be fed with host/device pointers and will move data automaticaly. ZFP operator requires matching data and compressor location.
  • Joined Array concept (besides Global Array and Local Array), which lets writers dump Local Arrays (no offsets no global shape) that are put together into a Global Array by the reader. One dimension of the arrays is selected for this join operation, while other dimensions must be the same for all writers. https://adios2.readthedocs.io/en/latest/components/components.html?highlight=Joined#shapes

File I/O

  • Default File engine is now BP5. If for some reason this causes problems, manually specify using "BP4" for your application.
  • BP5 is semantically identical to the staging engines, and any code that works with BP5 will work in situ without changes.
  • BP5 engine supports multithreaded reading to accelerate read performance for low-core counts.
  • BP5 Two level metadata aggregation and reduction reduced memory impact of collecting metadata and therefore is more scalable in terms of numbers of variables and writers than BP4.
  • Uses Blosc-2 instead of Blosc for lossless compression. The new compression operator is backward compatible with old files compressed with blosc. The name of the operator remains "blosc".

Staging

  • UCX dataplane added for SST staging engine to support networks under the UCX consortium
  • MPI dataplane added for SST staging engine. It relies on MPI intercommunicators to connect multiple independent MPI applications for staging purposes. Applications must enable multithreaded MPI for this dataplane.

Experimental features

  • Preliminary support for data structs. A struct can have single variables of basic types, and 1D fixed size arrays of basic types. Supported by BP5, SST and SSC engines.

What's Changed

Read more

Contributors

chuckatkins, vicentebolea, and 18 other contributors
Assets 2

ADIOS v2.9.0-rc1 Production Release

08 Feb 02:34
@vicentebolea vicentebolea
v2.9.0-rc1
c01b496
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
ADIOS v2.9.0-rc1 Production Release Pre-release
Pre-release 
Merge pull request #3476 from vicentebolea/bump-release-version

release: Bump version to v2.9.0-rc1
Assets 2

ADIOS v2.8.3 Production Patch Release

29 Jul 20:46
@chuckatkins chuckatkins
v2.8.3
473fe8c
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
ADIOS v2.8.3 Production Patch Release

This is a patch release that fixes two specific issues:

  • #3287 Allow the inline engine to work with ParaView Catalyst Live
  • #3291 Fix bpls issues with local arrays
Assets 2

ADIOS v2.8.2 Production Patch Release

13 Jul 00:57
@chuckatkins chuckatkins
v2.8.2
d244fdd
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
ADIOS v2.8.2 Production Patch Release

Summary

This is a patch release to fix a compatibility issue with BP4 and BLOSC compressed data arrays. It is essentially v2.8.1 with the addition of PR #3275.

Assets 2

ADIOS v2.8.1 Production Patch Release

02 Jun 06:32
@chuckatkins chuckatkins
v2.8.1
499d268
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
ADIOS v2.8.1 Production Patch Release

Summary

This is a patch release to address many of the issues identified with the v2.8.0 release after it was published.
For the new BP5 engine the fixes are

  • SelectSteps were always showing the first step in the file even if not selected
  • AppendAfterStep created a corrupted index file
  • bpls did not show the actual values of Global Values and Local Values (shown as 1D arrays)
  • bpls was not able to dump data of variables beyond the first step

Changes / Fixes

PR #3237
PR #3236
PR #3234
PR #3228
PR #3227
PR #3221
PR #3213
PR #3211
PR #3207
PR #3206
PR #3186
PR #3176
PR #3162
PR #3161
PR #3156
PR #3155
PR #3145
PR #3153

Assets 2

ADIOS v2.8.0 Production Release

29 Mar 11:54
@chuckatkins chuckatkins
v2.8.0
8121b20
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
ADIOS v2.8.0 Production Release

Summary

This is a major production release with a variety of new features and bug fixes.

Changes to the API

  • adios2::Mode::ReadRandomAccess mode is introduced for reading files with access to all steps. BeginStep/EndStep calls are NOT allowed. SetStepSelection() can be used to access specific steps in the file.
  • adios2::Mode::Read mode now requires using BeginStep/EndStep loop to access steps serially one by one. Variable inquiry fails outside BeginStep/EndStep sections. You need to modify your Open() statement to use the random-access mode if your code is accessing all steps in an existing file. Technically, this is only true for the new BP5 files, you can still read BP3/BP4/HDF5 files with your existing code but keep in mind that your code is not going to work when moving to BP5.
  • adios2::ADIOS::EnterComputationBlock(), adios2::ADIOS::ExitComputationBlock() are hints to ADIOS that a process is in a computing (i.e. non-communicating) phase. BP5 asynchronous I/O operations can schedule writing during such phases to avoid interfering with the application’s own communication.
  • GPU-aware I/O supports passing device-memory data pointers to the Put/Get functions, and ADIOS2 will automatically download/upload data from/to the device during I/O. Alternatively, an extra member function of the Variable class, SetMemorySpace(const adios2::MemorySpace mem) can explicitly tell ADIOS2 whether the pointer points to device memory or host memory.

Notable new features

BP5

BP5 is a new data format and there is a new engine, BP5, that writes and reads in this format. See https://adios2.readthedocs.io/en/latest/engines/engines.html for runtime options. There are four main advantages using this over the previous engine BP4.

  • Lower memory consumption. Deferred Puts use user buffer for I/O wherever possible thus saving on a memory copy. Aggregation uses a fixed-size shared-memory segment on each compute node instead of using MPI to send data from one process to another. Memory consumption can save 50% of extra memory usage compared to BP4 in some cases.
  • Faster metadata management improves write/read performance where hundreds or more variables are added to the output.
  • Improved functionality around appending many output steps into the same file. Better performance than writing new files each step. Restart can append to an existing series by truncating unwanted steps. Readers can filter out unwanted steps to only see and process a limited set of steps. Just like as in BP4, existing steps cannot be corrupted by appending new steps. Existing steps can be read by another application while new steps are written and the reader can wait for incoming steps, i.e., in situ processing through files works identically to using staging engines.
  • Asynchronous writing is an option with BP5, which can further improve performance in applications. This is a simple runtime option to turn it on/off, there is no need to tinker with I/O calls in the source code. This asynchronous I/O implementation does NOT need a multithreaded MPI library.

GPU Support

  • IO: ADIOS can now write and read data to and from CUDA device memory by passing a valid device pointer to the appropriate Put and Get API calls. Current support is limited to CUDA on NVIDIA GPUs but future releases will support both AMD and Intel GPUs.
  • Compression: The ZFP floating point compression operator has additional configurations options and now supports compression and decompression operations on NVIDIA GPUs for variable data on either host or device memory.

Further details on GPU-Aware I/O can be found in the ADIOS user guide

Plugin architecture for Engines and Operators

Engines and Operators can now be developed outside of ADIOS and loaded at runtime. For engine plugins, the engine should inherit from the PluginEngineInterface class. The engine library should also contain EngineCreate() and EngineDestroy() symbols that ADIOS will be able to use to create and destroy your engine. Similarly, Operator plugins should inherit from the PluginOperatorInterface class and the library should contain OperatorCreate() and OperatorDestroy() symbols. To help ADIOS find user plugin libraries, the ADIOS2_PLUGIN_PATH environment variable can be set. Further details on writing and using plugins can be found in the ADIOS user guide.

Compression operators backward-compatibility feature

From version 2.8.0 onward Operators manage their own metadata and their future versions will provide information on the (compression) libraries used in older datasets in case they cannot decompress the data with the current compression library it is built with. Users then may be able to rebuild the future ADIOS version with an older, compatible compression library that can read back their data. The downside of the redesign of Operators is that datasets written with lossy compression cannot be read back with ADIOS 2.8.0 and newer versions, so you need to build and use 2.7.1 for that.

Full Changelog: v2.7.1...v2.8.0

Assets 2