ADIOS v2.2.0 Performance Beta Release

Summary

This is the fourth pre-release of the ADaptable Input Output System, ADIOS, version 2 (v2.2.0). The primary focus of this beta release is increased performance through I/O aggregation. This version is not intended for production quality purposes.

Features

Major additions from the previous v2.1.1 Beta release include process aggregation and multi-block I/O.

Aggregation

So far ADIOS 2.x wrote one output file per each writing process. This provides good performance on parallel file systems for up to the point where the file system starts to struggle to handle too many files at once. Aggregation here means that a subset of processes collect the data from all the processes and only they write files. We have implemented the aggregation algorithm known from ADIOS 1.x with some modifications to use less memory than in ADIOS 1.x. At the API level, the user only needs to set the number of “substreams” different from the number of MPI processes. Figure 1 shows an example of how 4 processes can write to a single file. This implementation saturates the MPI network bandwidth while writing to disk in an asynchronous manner. In addition, the new modular architecture of ADIOS2 allows for future experimentation of different aggregation methods to chase performance on upcoming exascale systems.

Multi-Block Writing

ADIOS 2.x now allows writing independent variable pieces to can be later retrieved as a single selection. This is better illustrated in Figure 2, in which each sub-block of data, in blue, is written independently as it becomes available. Whereas a read request can be made for a certain selection, thus pulling data for multiple sub-blocks. This pattern is largely used to support experimental and observational data in which independent I/ O. This feature is naturally integrated into the ADIOS2 API, in which each Variable Put/Get block call is inserted in a container.

ADIOS v2.1.1 Staging Beta Release

This is the third pre-release of the ADaptable Input Output System, ADIOS, version 2 (v2.1.1). This is a beta release with staging support and a stable API intended for users and developers to begin integration and testing of ADIOS 2 into their applications, libraries, and workflows. This version is not intended for production quality purposes.

Major additions from the previous v2.1.0 Beta release include:

• SST Engine for staging data memory-to-memory across applications
• Updated Fortran, C and Python bindings APIs
• Several bugs corrected for default write and read BPFile engine
• Added option for flexible number of substreams (subfiles) based on MPI aggregation for NxM writes (N=mpi processes, M = user defined number of files)
• Added High-Level APIs to each supported language. These APIs are based on the language native IO (C++: adios2::fstream, C: adios2_FILE*, Fortran: file handler, Python: io stream object)
• User Guide documentation moved to readthedocs: http://adios2-adaptable-io-system-version-2.readthedocs.io/en/latest/

Feature	ADIOS 1.13 Support	ADIOS 2.1.1 Support
Write API
Local Variable	✅ (Offsets, LocalDimensions)	✅ (Start, Count)
Global Variable	✅ (Global)	✅ (Variable)
Joined Dimensions	✅	✅
Dimension Variable	✅	deprecated
Constant Dimension Variable	❌	✅
Selection	✅	partial
Memory Selection	✅	❌
Attributes	✅	✅
Transforms	✅	❌
Config XML	✅	✅
Threaded Operations	❌	✅
Time Aggregation	✅	✅
Collective Metadata	✅	✅
BP3 Format	✅	✅ (bitmap stats deprecated)
BP4 Format	---	❌
I/O Modes	✅ (Transports)	beta
BP3 SubFile N-to-N	✅ (POSIX)	✅
BP3 SubFile N-to-M	✅ (MPI_Aggregate)	✅ initial support
HDF5	✅	✅
WAN (DataMan)	❌	✅
Read API	✅	✅
Utils		all compatible with ADIOS1.12
bpmeta	✅	❌
bpls	✅	partial bpls2
bpdump	✅	❌
Language Support
C++	✅ through C	✅ native
Python	✅ Cython	✅ PyBind11
C	✅	✅
Fortran	✅	✅
Other
Automated testing	❌	✅ (via CTest and GoogleTest)
Windows support	❌	✅ (via CTest and GoogleTest)

ADIOS v2.1.0 Beta Release

This is the second pre-release of the ADaptable Input Output System, ADIOS, version 2 (v2.1.0). This is a beta release with a stable API intended for users and developers to begin integration and testing of ADIOS 2 into their applications, libraries, and workflows. This version is not intended for production quality purposes.

Major additions from the v2.0.0 Alpha release include:

• Re-designed Read and Write APIs for uniformity between the two
• Native read and write support for the ADIOS BP format
• Read and write support for parallel and serial HDF5
• Full support for C, Fortran, and Python APIs in addition to the native C++ interface.
• Initial staging implementations with SST for HPC environments and DataMan for Wide Area Networks.

Feature	ADIOS 1.12 Support	ADIOS 2.1.0 Support
Write API
Local Variable	✅ (Offsets, LocalDimensions)	✅ (Start, Count)
Global Variable	✅ (Global)	✅ (Variable)
Joined Dimensions	❌	✅
Dimension Variable	✅	deprecated
Constant Dimension Variable	❌	✅
Selection	✅	partial
Memory Selection	✅	❌
Attributes	✅	✅
Transforms	✅	❌
Config XML	✅	✅
Threaded Operations	❌	✅
Time Aggregation	✅	✅
Collective Metadata	✅	✅
BP1 Format	✅	✅ (bitmap stats deprecated)
BP2 Format	---	❌
I/O Modes	✅ (Transports)	partial
BP1 File N-to-N	✅ (POSIX)	✅
BP1 File N-to-M	✅ (MPI_Aggregate)	❌
HDF5	✅	✅
WAN (DataMan)	❌	✅
Read API	✅	✅
Utils		all compatible with ADIOS1.12
bpmeta	✅	❌
bpls	✅	partial bpls2
bpdump	✅	❌
Language Support
C++	✅ through C	✅ native
Python	✅ Cython	✅ PyBind11
C	✅	✅
Fortran	✅	✅ Still to do
Other
Automated testing	❌	✅ (via CTest and GoogleTest)
Windows support	❌	✅ (via CTest and GoogleTest)

ADIOS v2.0.0 Alpha Release

Initial release of the ADaptable Input Output System, ADIOS, version 2.0.0. This is an alpha release intended to be distributed so users and developers can test the preliminary list of supported features, provide feedback on the source code, documentation, API, and development flows, etc., and report bugs. This version is not intended for production quality purposes.

Please see Getting Started for instructions on how to configure and build this release, along with the User Guide.

Summary
Supported 64-bit OS	Linux
	Mac OS X El Capitan
	Windows 10
Minimum Compiler Versions	gcc v4.8.1
	clang 3.3
	Visual Studio 2015
	Intel 16
Distributed 3rd Party	PyBind11
	pugixml
	GoogleTest
	KWSys
	JSON for Modern C++

Feature Support

This is list of partial features support in ADIOS 2.0.0 that are included in ADIOS 1.12. The goal of this list is to provide a simple overview to those familiar with ADIOS. It also provides a roadmap of the features from ADIOS 1.12 to be implemented in the next beta release in December 2017.

Feature	ADIOS 1.12 Support	ADIOS 2.0.0 Support
Write API
Local Variable	✅ (Offsets, LocalDimensions)	✅ (Start, Count)
Global Variable	✅ (Global)	✅ (Variable)
Joined Dimensions	❌	✅
Dimension Variable	✅	deprecated
Constant Dimension Variable	❌	✅
Selection	✅	partial
Memory Selection	✅	❌
Attributes	✅	❌
Transforms	✅	❌
Config XML	✅	✅
Threaded Operations	❌	✅
Time Aggregation	✅	✅
Collective Metadata	✅	❌ (see Utils bpmeta)
BP1 Format	✅	✅ (bitmap stats deprecated)
BP2 Format	---	(experimental)
I/O Modes	✅ (Transports)	partial
BP1 File N-to-N	✅ (POSIX)	✅
BP1 File N-to-M	✅ (MPI_Aggregate)	❌
HDF5	✅	✅
WAN (DataMan)	❌	✅
Read API	✅	partial wrapper around ADIOS1
Utils		all compatible with ADIOS1.12
bpmeta	✅	❌
bpls	✅	❌
bpdump	✅	❌
Language Support
C++	✅ through C	✅ native
Python	✅ Cython	✅ PyBind11
C	✅	❌
Fortran	✅	❌
Other
Automated testing	❌	✅ (via CTest and GoogleTest)
Windows support	❌	✅ (via CTest and GoogleTest)