Skip to content

Commit

Permalink
Merge remote-tracking branch 'origin/adios-xrootd' into adios-xrootd
Browse files Browse the repository at this point in the history
* origin/adios-xrootd: (164 commits)
  Fixes for FreeBSD, including upstream (ornladios#4138)
  Bump version to v2.10.0
  Setting the derived variable support OFF by default
  Add the CURL function to derived variables (ornladios#4114)
  Add -f file1 [file2...] option to process adios files from a list instead of a campaign recording
  DataPlane Configuration changes (ornladios#4121)
  update doc
  Add attribute support to campaign
  Add a random string to each database name to avoid collision when running multiple applications in the same directory at the same time. Fixes issues with CI that runs ctest in parallel
  Initialize ADIOS::m_UserOption before tentaively calling ProcessUserConfig()
  - set ACA version to  0.1 - remove debug prints - add doc on Campaign Management in Advanced section - change static struct of UserOptions to class member of ADIOS class to make it work with gcc 4.8
  used size_t not int for map indexing to avoid type conversion
  fix remote server test, use the new binary name
  clang-format
  Use the name of the campaign in the cache path to avoid name collision
  rename remote_server to adios2_remote_server
  bug fix: the order of entries in bpdataset table is undefined but the campaign data reader relied on calculating the index as if it was sorted by the insertion order. Use a map instead to store the rowid and use that as index for the bpfile elements.
  Use yaml parser in campaign manager python script
  change a long variable to int64_t to avoid size confusion on windows
  do not include unistd.h
  Fix compiler error Remove extra file not needed
  Add support for user options in ~/.config/adios2/adios2.yaml Currently supported options:
  cmake: add sqlite3 and zlib dep in adios2 cmake pkg
  Different names for MPI and Serial tests (ornladios#4118)
  EVpath upstream to make NO_RDMA more robust (ornladios#4116)
  Warnings (ornladios#4113)
  Don't use assert() in tests (ornladios#4108)
  - Only add campaign store to file name if that is not absolute path - list command supports second argument as path
  flake8 fixes
  Update campaign manager script to handle config file, time in nanosecond format, and avoiding conflict when updating database
  dill 2024-03-12 (ebc98c4d) (ornladios#4091)
  Don't run derived test in MPI mode, it's not written for that (ornladios#4104)
  Fix static blosc2 build (ornladios#4093)
  ci: add ccache job summary (ornladios#4101)
  Fix typo in fortran.rst (ornladios#4102)
  WIP: Make Fortran tests fail with a non-zero exit code (ornladios#4097)
  Bison 3.8 Parser (ornladios#4062)
  Do not create adios-campaign/ unless there is something to record
  Add setup for Aurora (load adios2 as e4s package)
  Completely hide derived variables in C API if not enabled. Print warning inside Fortran F2C function.
  adios2_define_derived_variable C/Fortran API. C is compiled conditionally, like the C++ API. The Fortran function is always available, it will print a WARNING if derived variable is not supported. Added Fortran test for magnitude().
  Fix links to tutorial materials (ornladios#4086)
  BlockIndex.Evaluate() made branch based on BP5 vs BP4. To support CampaignReader engine, decision is made based on whether MinBlocksInfo is supported by engine.
  Update documentation for 2.10 changes to the GPU-backend (ornladios#4083)
  Add test for single string attribute vs string array attribute with a single element
  - Python: fix for scalar reading. If a global value has 1 step (i.e. always in streaming), read returns a 0-dim numpy array (single value). If the variable has multiple steps (only in ReadRandomAccess mode), read returns a 1-dim numpy array even if the step selection is a single step. This way, read of a certain variable always results in the same type of array no matter the number of steps selected. - Python: fix for string attributes: return a string, not a list of one element which is a string, to be consistent with string global values and with other APIs.
  format more
  format
  Python: add the same treatment to attributes as to variables before: return scalars (0-dim ndarray) for single value attributes.
  Raise an exception if remote open fails (ornladios#4069)
  Fortran bindings for memory space related functions (ornladios#4077)
  consolidate (ornladios#4078)
  Making the Detect memory space available regardless of the backend used
  Testing code for the C bindings with memory space API
  Adding c bindings for getting the shape of a variable based on a memory space
  Adding c bindings for setting and getting the memory space
  Small typo fixes
  - Restructure python API doc in separate main topic, add working example to it. - What's new for 2.10 - Usage on DOE machines
  Fix Reord to use MinBlocksInfo where appropriate (ornladios#4071)
  Using the correct flag to detect the CUDA backend in ZFP
  clang-format fix
  fixed warning
  added support to read back from H5T_STRING VARIABLES it turns out strings written out through h5py are all variable strings
  format
  fixes to still be able to build with gcc 4.8.2. Needed for OLCF DTN nodes
  Add minmax and shape functions to CampaignReader, so that per-block info is complete when listing campaign archives
  do not flush io and adios in read mode. BP5 reader does not like it.
  Campaign engine is recognized by file extension.
  WIP. Changed the name of the campaign config file.
  Added reading of configuration parametyers from ./config/adios2
  Use GetEstimatedSize in encryption operator plugin
  Allow plugin operators to take advantage of the estimated size API
  Setting the derived variable support OFF by default
  Add the CURL function to derived variables (ornladios#4114)
  Add -f file1 [file2...] option to process adios files from a list instead of a campaign recording
  DataPlane Configuration changes (ornladios#4121)
  update doc
  Add attribute support to campaign
  Add a random string to each database name to avoid collision when running multiple applications in the same directory at the same time. Fixes issues with CI that runs ctest in parallel
  Initialize ADIOS::m_UserOption before tentaively calling ProcessUserConfig()
  - set ACA version to  0.1 - remove debug prints - add doc on Campaign Management in Advanced section - change static struct of UserOptions to class member of ADIOS class to make it work with gcc 4.8
  used size_t not int for map indexing to avoid type conversion
  fix remote server test, use the new binary name
  clang-format
  Use the name of the campaign in the cache path to avoid name collision
  rename remote_server to adios2_remote_server
  bug fix: the order of entries in bpdataset table is undefined but the campaign data reader relied on calculating the index as if it was sorted by the insertion order. Use a map instead to store the rowid and use that as index for the bpfile elements.
  Use yaml parser in campaign manager python script
  change a long variable to int64_t to avoid size confusion on windows
  do not include unistd.h
  Fix compiler error Remove extra file not needed
  Add support for user options in ~/.config/adios2/adios2.yaml Currently supported options:
  cmake: add sqlite3 and zlib dep in adios2 cmake pkg
  Different names for MPI and Serial tests (ornladios#4118)
  EVpath upstream to make NO_RDMA more robust (ornladios#4116)
  Warnings (ornladios#4113)
  Don't use assert() in tests (ornladios#4108)
  - Only add campaign store to file name if that is not absolute path - list command supports second argument as path
  flake8 fixes
  Update campaign manager script to handle config file, time in nanosecond format, and avoiding conflict when updating database
  ...
  • Loading branch information
dmitry-ganyushin committed Apr 16, 2024
2 parents ba3a41e + 5b1a1a0 commit 3c96391
Show file tree
Hide file tree
Showing 221 changed files with 10,699 additions and 6,456 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/everything.yml
Expand Up @@ -188,7 +188,7 @@ jobs:
- name: Build
run: gha/scripts/ci/gh-actions/run.sh build
- name: Print ccache statistics
run: ccache -s
run: ccache -s | tee $GITHUB_STEP_SUMMARY
- name: Save cache
uses: actions/cache/save@v3
if: ${{ github.ref_name == 'master' && steps.restore-cache.outputs.cache-hit != 'true' }}
Expand Down
20 changes: 16 additions & 4 deletions CMakeLists.txt
Expand Up @@ -183,7 +183,7 @@ adios_option(Profiling "Enable support for profiling" AUTO)
adios_option(Endian_Reverse "Enable support for Little/Big Endian Interoperability" AUTO)
adios_option(Sodium "Enable support for Sodium for encryption" AUTO)
adios_option(Catalyst "Enable support for in situ visualization plugin using ParaView Catalyst" AUTO)
adios_option(Campaign "Enable support for Campaigns (requires SQLite3 and ZLIB)" OFF)
adios_option(Campaign "Enable support for Campaigns (requires SQLite3 and ZLIB)" AUTO)
adios_option(AWSSDK "Enable support for S3 compatible storage using AWS SDK's S3 module" OFF)
adios_option(Derived_Variable "Enable support for derived variables" OFF)
adios_option(PIP "Enable support for pip packaging" OFF)
Expand Down Expand Up @@ -444,9 +444,21 @@ foreach(opt IN LISTS ADIOS2_CONFIG_OPTS)
endif()
endforeach()

if (ADIOS2_HAVE_SST AND (ADIOS2_SST_HAVE_LIBFABRIC OR ADIOS2_SST_HAVE_UCX))
message(" RDMA Transport for Staging: Available")
set (HPCDataPlaneList "")
if (ADIOS2_HAVE_SST)
if (ADIOS2_SST_HAVE_LIBFABRIC)
set (HPCDataPlaneList "${HPCDataPlaneList} fabric")
endif()
if (ADIOS2_SST_HAVE_UCX)
set (HPCDataPlaneList "${HPCDataPlaneList} UCX")
endif()
if (ADIOS2_SST_HAVE_MPI_DP)
set (HPCDataPlaneList "${HPCDataPlaneList} MPI")
endif()
endif()
if ({HPCDataPlaneList} STREQUAL "")
message(" Possible RDMA DataPlanes for SST: <none>")
else()
message(" RDMA Transport for Staging: Unconfigured")
message(" Possible RDMA DataPlanes for SST: ${HPCDataPlaneList}")
endif()

37 changes: 37 additions & 0 deletions bindings/C/adios2/c/adios2_c_io.cpp
Expand Up @@ -150,6 +150,43 @@ adios2_error adios2_set_transport_parameter(adios2_io *io, const size_t transpor
}
}

#ifdef ADIOS2_HAVE_DERIVED_VARIABLE
adios2_derived_variable *adios2_define_derived_variable(adios2_io *io, const char *name,
const char *expression,
const adios2_derived_var_type type)
{
adios2_derived_variable *variable = nullptr;

try
{
adios2::helper::CheckForNullptr(io, "for adios2_io, in call to adios2_define_variable");
adios2::core::IO &ioCpp = *reinterpret_cast<adios2::core::IO *>(io);
adios2::DerivedVarType typeCpp = adios2::DerivedVarType::MetadataOnly;
switch (type)
{
case adios2_derived_var_type_metadata_only:
typeCpp = adios2::DerivedVarType::MetadataOnly;
break;
case adios2_derived_var_type_expression_string:
typeCpp = adios2::DerivedVarType::ExpressionString;
break;
case adios2_derived_var_type_store_data:
typeCpp = adios2::DerivedVarType::StoreData;
break;
}
adios2::core::VariableDerived *variableCpp = nullptr;
variableCpp = &ioCpp.DefineDerivedVariable(name, expression, typeCpp);
variable = reinterpret_cast<adios2_derived_variable *>(variableCpp);
}
catch (...)
{

adios2::helper::ExceptionToError("adios2_define_variable");
}
return variable;
}
#endif

adios2_variable *adios2_define_variable(adios2_io *io, const char *name, const adios2_type type,
const size_t ndims, const size_t *shape,
const size_t *start, const size_t *count,
Expand Down
20 changes: 20 additions & 0 deletions bindings/C/adios2/c/adios2_c_io.h
Expand Up @@ -122,6 +122,26 @@ adios2_variable *adios2_define_variable(adios2_io *io, const char *name, const a
const size_t *start, const size_t *count,
const adios2_constant_dims constant_dims);

#ifdef ADIOS2_HAVE_DERIVED_VARIABLE
/**
* @brief Define a derived variable within io
* @param io handler that owns the variable
* @param name unique variable identifier
* @param type primitive type from enum adios2_type in adios2_c_types.h
* @param ndims number of dimensions
* @param shape global dimension
* @param start local offset
* @param count local dimension
* @param constant_dims adios2_constant_dims_true:: shape, start, count
* won't change; adios2_constant_dims_false: shape, start, count will change
* after definition
* @return success: handler, failure: NULL
*/
adios2_derived_variable *adios2_define_derived_variable(adios2_io *io, const char *name,
const char *expression,
const adios2_derived_var_type type);
#endif

/**
* @brief Retrieve a variable handler within current io handler
* @param io handler to variable io owner
Expand Down
11 changes: 11 additions & 0 deletions bindings/C/adios2/c/adios2_c_types.h
Expand Up @@ -23,6 +23,7 @@ extern "C" {
typedef struct adios2_adios adios2_adios;
typedef struct adios2_io adios2_io;
typedef struct adios2_variable adios2_variable;
typedef struct adios2_derived_variable adios2_derived_variable;
typedef struct adios2_attribute adios2_attribute;
typedef struct adios2_engine adios2_engine;
typedef struct adios2_operator adios2_operator;
Expand Down Expand Up @@ -139,6 +140,16 @@ typedef enum
adios2_arrayordering_auto
} adios2_arrayordering;

#ifdef ADIOS2_HAVE_DERIVED_VARIABLE
/** Type of derived variables */
typedef enum
{
adios2_derived_var_type_metadata_only = 0,
adios2_derived_var_type_expression_string = 1,
adios2_derived_var_type_store_data = 2
} adios2_derived_var_type;
#endif

static const size_t adios2_string_array_element_max_size = 4096;

static const size_t adios2_local_value_dim = SIZE_MAX - 2;
Expand Down
8 changes: 0 additions & 8 deletions bindings/C/adios2/c/adios2_c_variable.cpp
Expand Up @@ -79,11 +79,7 @@ adios2_error adios2_set_shape(adios2_variable *variable, const size_t ndims, con

adios2::MemorySpace adios2_ToMemorySpace(const adios2_memory_space Cmem)
{
#ifdef ADIOS2_HAVE_GPU_SUPPORT
adios2::MemorySpace mem = adios2::MemorySpace::Detect;
#else
adios2::MemorySpace mem = adios2::MemorySpace::Host;
#endif
switch (Cmem)
{

Expand All @@ -104,11 +100,7 @@ adios2::MemorySpace adios2_ToMemorySpace(const adios2_memory_space Cmem)

adios2_memory_space adios2_FromMemorySpace(const adios2::MemorySpace mem)
{
#ifdef ADIOS2_HAVE_GPU_SUPPORT
adios2_memory_space Cmem = adios2_memory_space_detect;
#else
adios2_memory_space Cmem = adios2_memory_space_host;
#endif
switch (mem)
{

Expand Down
2 changes: 2 additions & 0 deletions bindings/Fortran/CMakeLists.txt
Expand Up @@ -44,6 +44,7 @@ add_library(adios2_fortran
modules/adios2_io_open_mod.F90
modules/adios2_io_open_serial_smod.F90
modules/adios2_io_define_variable_mod.f90
modules/adios2_io_define_derived_variable_mod.f90
modules/adios2_io_define_attribute_mod.f90
modules/adios2_engine_mod.f90
modules/adios2_engine_begin_step_mod.f90
Expand All @@ -54,6 +55,7 @@ add_library(adios2_fortran
modules/adios2_variable_max_mod.f90
modules/adios2_operator_mod.f90
)

set_property(TARGET adios2_fortran PROPERTY EXPORT_NAME fortran)
set_property(TARGET adios2_fortran PROPERTY OUTPUT_NAME adios2${ADIOS2_LIBRARY_SUFFIX}_fortran)

Expand Down
23 changes: 22 additions & 1 deletion bindings/Fortran/f2c/adios2_f2c_io.cpp
Expand Up @@ -201,6 +201,27 @@ void FC_GLOBAL(adios2_define_variable_f2c,
}
}

void FC_GLOBAL(adios2_define_derived_variable_f2c,
ADIOS2_DEFINE_DERIVED_VARIABLE_F2C)(adios2_derived_variable **variable,
adios2_io **io, const char *name,
const char *expression, const int *type,
int *ierr)
{
#ifdef ADIOS2_HAVE_DERIVED_VARIABLE
*variable = adios2_define_derived_variable(*io, name, expression,
static_cast<adios2_derived_var_type>(*type));
*ierr = (*variable == NULL) ? static_cast<int>(adios2_error_exception)
: static_cast<int>(adios2_error_none);
#else
std::cout << "ADIOS2 Warning: adios2_define_derived_variable() is not supported in the "
"current ADIOS2 build. The expression "
<< expression << " will be ignored and the variable " << name
<< " will not be produced." << std::endl;
*variable = nullptr;
*ierr = static_cast<int>(adios2_error_exception);
#endif
}

struct cnamelist
{
char **names;
Expand Down Expand Up @@ -267,7 +288,7 @@ void FC_GLOBAL(adios2_retrieve_namelist_f2c,
len = static_cast<size_t>(namelist_len);
}
// copy C string without '\0'
strncpy(fs, info->names[i], len);
memcpy(fs, info->names[i], len);
// pad with spaces
memset(fs + len, ' ', namelist_len - len);
}
Expand Down
40 changes: 40 additions & 0 deletions bindings/Fortran/modules/adios2_io_define_derived_variable_mod.f90
@@ -0,0 +1,40 @@
!
! Distributed under the OSI-approved Apache License, Version 2.0. See
! accompanying file Copyright.txt for details.
!
! adios2_io_define_variable_mod.f90 : ADIOS2 Fortran bindings for IO class
! overloaded (C++ template) function adios2_define_variable
!
! Created on: Mar 13, 2017
! Author: William F Godoy godoywf@ornl.gov
!

module adios2_io_define_derived_variable_mod
use adios2_parameters_mod
implicit none

external adios2_define_derived_variable_f2c

contains

subroutine adios2_define_derived_variable(variable, io, name, expression, adios2_derived_var_type, &
ierr)
type(adios2_derived_variable), intent(out) :: variable
type(adios2_io), intent(in) :: io
character*(*), intent(in) :: name
character*(*), intent(in) :: expression
integer, intent(in):: adios2_derived_var_type
integer, intent(out) :: ierr

call adios2_define_derived_variable_f2c(variable%f2c, io%f2c, &
TRIM(ADJUSTL(name))//char(0), TRIM(ADJUSTL(expression))//char(0), &
adios2_derived_var_type, ierr)
if( ierr == 0 ) then
variable%valid = .true.
variable%name = name
variable%type = adios2_derived_var_type
end if

end subroutine

end module

0 comments on commit 3c96391

Please sign in to comment.