TileDB 2.18.0

TileDB v2.18.0 Release Notes

Announcements

• TileDB 2.18, targeted for release in November 2023, includes a preview set of aggregate pushdown APIs. The APIs will be finalized in 2.19 with performance improvements.

Disk Format

• Fix the format specification for group members. #4380
• Update fragment format spec for info on tile sizes and tile offsets. #4416

Configuration changes

• Remove vfs.file.max_parallel_ops config option. #3964

Breaking C API changes

• Behavior breaking change: tiledb_group_remove_member cannot remove named members by URI if the URI is duplicated. #4391

New features

• Support custom headers on s3 requests. #4400
• Add support for Google Cloud Storage on Windows. #4138
• Query aggregates REST support. #4415
• Add random number generator global singleton. #4437

Improvements

• Add an aspect template argument to class CAPIFunction. #4430
• Fix enumeration value not found error message. #4339
• Allow forward slashes in enumeration names. #4454
• Adding extra in memory offset for var tiles on read path. #4294
• Aggregators: adding mean aggregate. #4292
• Remove parallel file I/O. #3964
• C.41 constructor: S3. #4368
• Changed the logic of the main loop in Domain::cell_order_cmp_impl<char>. #3847
• Add REST client support for delete_fragments and delete_fragments_list. #3923
• Remove the internal ValidityVector::init_bytemap method. #4071
• Use exceptions for error handling in the Metadata code. #4012
• Move min/max computation to aggregate_with_count. #4394
• Add memory tracking to Enumeration loading. #4395
• Logs for nested exceptions now incorporate all messages. #4393
• Implement read logging modes. #4386
• Move counts computation to aggregate_with_count. #4401
• Fix delete_fragments handlers according to new model. #4414
• Aggregates: change to use size instead of min cell/max cell. #4436
• Allow for empty enumerations. #4423
• Aggregates: combine sum/mean aggregators. #4440

Documentation

• Update docs for set data buffer APIs. #4381

Defects removed

• Fix serialization of empty enumerations. #4472
• Patch curl to fix CVE-2023-38545. #4408
• Update libwebp to version 1.3.2 to fix CVE-2023-5129. #4384
• Fix missing WebP::sharpyuv installation command. #4325
• Fix re-loading enumerations from the REST server. #4335
• Save error on the context before returning error code in vfs_open. #4347
• Filter pipeline: fix coord filter type. #4351
• Fix creation of Enumerations with std::vector<bool>. #4362
• Revert change to S3 is_object, is_bucket checks. #4370
• Specify whether the HDFS backend is enabled in the string returned by tiledb_as_built_dump. #4364
• Fix fragment corruption in the GlobalOrderWriter. #4383
• Fix bug in FragmentMetadata::get_footer_size. #4212
• Global order writer: don't try to delete the current fragment if it had failed to be created. #4052
• Fix bug in ordered label reader for unsigned integers. #4404
• Add missing include object.h in group_experimental.h. #4411
• Support Azure Storage SAS tokens that don't start with a question mark. #4418
• Avoid empty Enumeration REST requests. #4369
• Make the initializer_list overload of Subarray::set_subarray zero-copy. #4062

API changes

C API

• New: Aggregate pushdown C API. #4176
• Implement tiledb_handle_load_array_schema_request. #4399
• Add ArraySchemaEvolution::extend_enumeration. #4445
• Allow null cfg in group consolidate_metadata and vacuum_metadata. #4412

C++ API

• Fix UBSAN error in the Enumeration CPP API. #4357
• Remove default operation type from query condition set membership API (introduced in 2.17). #4390
• Fix CPP set buffer APIs to work with aggregates. #4434
• Adding CPP example for consolidation plan. #4433
• New: Aggregate API for MEAN and NULL_COUNT capi/cppapi wraps. #4443

Build System Changes

• Fix static linking failures and build ExampleExe_static on Windows. #4336
• Run Windows nightly debug build on C:/ drive to avoid out of space errors. #4352
• Manage dependencies with vcpkg by default. #4348
• Find the OpenSSL dependency of the AWS SDK on non-Windows only. #4359
• Use generator expressions when setting compile options for the benchmarks. #4328
• Bump default compiler to GCC 10 and macos images from 11 to 12. #4367
• Fix and optimize the Release CI workflow. #4373
• Stop building gRPC when GCS is enabled. #4435
• Require only ucrt build from Rtools. #4361
• Update the AWS SDK to version 1.11.160. #4214
• Remove warnings from benchmarks. #4396
• Update clang-format to version 16. #4397
• Update the Google Cloud Storage SDK to its latest version (2.15.1). #4031
• Update capnproto to version 1.0.1. #4405
• Fix compilation with GCC 13. #4331
• Compilation fix for GCC 9.4 of Ubuntu 20.04. #4355
• Don't error out if DELETE symbol is defined. #4365

Test only changes

• Aggregates pushdown: adding tests for var size buffer overflow. #4315
• Aggregates pushdown: adding incomplete tests. #4301
• Add group backwards compatibility tests. #3996
• Aggregates: Adding tests for computation policies. #4421
• Aggregators: adding benchmark for aggregate_with_count class. #4420

Experimental Changes

• Add support for basic mimo node to TileDB task graph. #3977
• Verify that dependency-only edges using std::monostate work with nodes and edges. #3939

TileDB 2.18.0-rc3

TileDB v2.18.0 Release Notes

Announcements

• TileDB 2.18, targeted for release in November 2023, includes a preview set of aggregate pushdown APIs. The APIs will be finalized in 2.19 with performance improvements.

Disk Format

• Fix the format specification for group members. #4380
• Update fragment format spec for info on tile sizes and tile offsets. #4416

Configuration changes

• Remove vfs.file.max_parallel_ops config option. #3964

Breaking C API changes

• Behavior breaking change: tiledb_group_remove_member cannot remove named members by URI. #4391

New features

• Support custom headers on s3 requests. #4400
• Add support for Google Cloud Storage on Windows. #4138
• Query aggregates REST support. #4415
• Add random number generator global singleton. #4437

Improvements

• Add an aspect template argument to class CAPIFunction. #4430
• Fix enumeration value not found error message. #4339
• Allow forward slashes in enumeration names. #4454
• Adding extra in memory offset for var tiles on read path. #4294
• Aggregators: adding mean aggregate. #4292
• Remove parallel file I/O. #3964
• C.41 constructor: S3. #4368
• Changed the logic of the main loop in Domain::cell_order_cmp_impl<char>. #3847
• Add REST client support for delete_fragments and delete_fragments_list. #3923
• Remove the internal ValidityVector::init_bytemap method. #4071
• Use exceptions for error handling in the Metadata code. #4012
• Move min/max computation to aggregate_with_count. #4394
• Add memory tracking to Enumeration loading. #4395
• Logs for nested exceptions now incorporate all messages. #4393
• Implement read logging modes. #4386
• Move counts computation to aggregate_with_count. #4401
• Fix delete_fragments handlers according to new model. #4414
• Aggregates: change to use size instead of min cell/max cell. #4436
• Allow for empty enumerations. #4423
• Aggregates: combine sum/mean aggregators. #4440

Documentation

• Update docs for set data buffer APIs. #4381

Defects removed

• Fix serialization of empty enumerations. #4472
• Patch curl to fix CVE-2023-38545. #4408
• Update libwebp to version 1.3.2 to fix CVE-2023-5129. #4384
• Fix missing WebP::sharpyuv installation command. #4325
• Fix re-loading enumerations from the REST server. #4335
• Save error on the context before returning error code in vfs_open. #4347
• Filter pipeline: fix coord filter type. #4351
• Fix creation of Enumerations with std::vector<bool>. #4362
• Revert change to S3 is_object, is_bucket checks. #4370
• Specify whether the HDFS backend is enabled in the string returned by tiledb_as_built_dump. #4364
• Fix fragment corruption in the GlobalOrderWriter. #4383
• Fix bug in FragmentMetadata::get_footer_size. #4212
• Global order writer: don't try to delete the current fragment if it had failed to be created. #4052
• Fix bug in ordered label reader for unsigned integers. #4404
• Add missing include object.h in group_experimental.h. #4411
• Support Azure Storage SAS tokens that don't start with a question mark. #4418
• Avoid empty Enumeration REST requests. #4369
• Make the initializer_list overload of Subarray::set_subarray zero-copy. #4062

API changes

C API

• New: Aggregate pushdown C API. #4176
• Implement tiledb_handle_load_array_schema_request. #4399
• Add ArraySchemaEvolution::extend_enumeration. #4445
• Allow null cfg in group consolidate_metadata and vacuum_metadata. #4412

C++ API

• Fix UBSAN error in the Enumeration CPP API. #4357
• Remove default operation type from query condition set membership API (introduced in 2.17). #4390
• Fix CPP set buffer APIs to work with aggregates. #4434
• Adding CPP example for consolidation plan. #4433
• New: Aggregate API for MEAN and NULL_COUNT capi/cppapi wraps. #4443

Build System Changes

• Fix static linking failures and build ExampleExe_static on Windows. #4336
• Run Windows nightly debug build on C:/ drive to avoid out of space errors. #4352
• Manage dependencies with vcpkg by default. #4348
• Find the OpenSSL dependency of the AWS SDK on non-Windows only. #4359
• Use generator expressions when setting compile options for the benchmarks. #4328
• Bump default compiler to GCC 10 and macos images from 11 to 12. #4367
• Fix and optimize the Release CI workflow. #4373
• Stop building gRPC when GCS is enabled. #4435
• Require only ucrt build from Rtools. #4361
• Update the AWS SDK to version 1.11.160. #4214
• Remove warnings from benchmarks. #4396
• Update clang-format to version 16. #4397
• Update the Google Cloud Storage SDK to its latest version (2.15.1). #4031
• Update capnproto to version 1.0.1. #4405
• Fix compilation with GCC 13. #4331
• Compilation fix for GCC 9.4 of Ubuntu 20.04. #4355
• Don't error out if DELETE symbol is defined. #4365

Test only changes

• Aggregates pushdown: adding tests for var size buffer overflow. #4315
• Aggregates pushdown: adding incomplete tests. #4301
• Add group backwards compatibility tests. #3996
• Aggregates: Adding tests for computation policies. #4421
• Aggregators: adding benchmark for aggregate_with_count class. #4420

Experimental Changes

• Add support for basic mimo node to TileDB task graph. #3977
• Verify that dependency-only edges using std::monostate work with nodes and edges. #3939

TileDB 2.18.0-rc2

TileDB v2.18.0 Release Notes

Announcements

• TileDB 2.18, targeted for release in November 2023, includes an internal release of a preview set of aggregate pushdown APIs. The APIs will be finalized in 2.19.

Disk Format

• Fix the format specification for group members. #4380
• Update fragment format spec for info on tile sizes and tile offsets. #4416

Configuration changes

• Remove vfs.file.max_parallel_ops config option. #3964

Breaking C API changes

• Behavior breaking change: tiledb_group_remove_member cannot remove named members by URI. #4391

New features

• Support custom headers on s3 requests. #4400
• Add support for Google Cloud Storage on Windows. #4138
• Query aggregates REST support. #4415
• Add random number generator global singleton. #4437

Improvements

• Add an aspect template argument to class CAPIFunction. #4430
• Fix enumeration value not found error message. #4339
• Allow forward slashes in enumeration names. #4454
• Adding extra in memory offset for var tiles on read path. #4294
• Aggregators: adding mean aggregate. #4292
• Remove parallel file I/O. #3964
• C.41 constructor: S3. #4368
• Changed the logic of the main loop in Domain::cell_order_cmp_impl<char>. #3847
• Add REST client support for delete_fragments and delete_fragments_list. #3923
• Remove the internal ValidityVector::init_bytemap method. #4071
• Use exceptions for error handling in the Metadata code. #4012
• Move min/max computation to aggregate_with_count. #4394
• Add memory tracking to Enumeration loading. #4395
• Logs for nested exceptions now incorporate all messages. #4393
• Implement read logging modes. #4386
• Move counts computation to aggregate_with_count. #4401
• Fix delete_fragments handlers according to new model. #4414
• Aggregates: change to use size instead of min cell/max cell. #4436
• Allow for empty enumerations. #4423
• Aggregates: combine sum/mean aggregators. #4440

Documentation

• Update docs for set data buffer APIs. #4381

Defects removed

• Fix serialization of empty enumerations. #4472
• Patch curl to fix CVE-2023-38545. #4408
• Update libwebp to version 1.3.2 to fix CVE-2023-5129. #4384
• Fix missing WebP::sharpyuv installation command. #4325
• Fix re-loading enumerations from the REST server. #4335
• Save error on the context before returning error code in vfs_open. #4347
• Filter pipeline: fix coord filter type. #4351
• Fix creation of Enumerations with std::vector<bool>. #4362
• Revert change to S3 is_object, is_bucket checks. #4370
• Specify whether the HDFS backend is enabled in the string returned by tiledb_as_built_dump. #4364
• Fix fragment corruption in the GlobalOrderWriter. #4383
• Fix bug in FragmentMetadata::get_footer_size. #4212
• Global order writer: don't try to delete the current fragment if it had failed to be created. #4052
• Fix bug in ordered label reader for unsigned integers. #4404
• Add missing include object.h in group_experimental.h. #4411
• Support Azure Storage SAS tokens that don't start with a question mark. #4418
• Avoid empty Enumeration REST requests. #4369
• Make the initializer_list overload of Subarray::set_subarray zero-copy. #4062

API changes

C API

• New: Aggregate pushdown C API. #4176
• Implement tiledb_handle_load_array_schema_request. #4399
• Add ArraySchemaEvolution::extend_enumeration. #4445
• Allow null cfg in group consolidate_metadata and vacuum_metadata. #4412

C++ API

• Fix UBSAN error in the Enumeration CPP API. #4357
• Remove default operation type from query condition set membership API (introduced in 2.17). #4390
• Fix CPP set buffer APIs to work with aggregates. #4434
• Adding CPP example for consolidation plan. #4433
• New: Aggregate API for MEAN and NULL_COUNT capi/cppapi wraps. #4443

Build System Changes

• Fix static linking failures and build ExampleExe_static on Windows. #4336
• Run Windows nightly debug build on C:/ drive to avoid out of space errors. #4352
• Manage dependencies with vcpkg by default. #4348
• Find the OpenSSL dependency of the AWS SDK on non-Windows only. #4359
• Use generator expressions when setting compile options for the benchmarks. #4328
• Bump default compiler to GCC 10 and macos images from 11 to 12. #4367
• Fix and optimize the Release CI workflow. #4373
• Stop building gRPC when GCS is enabled. #4435
• Require only ucrt build from Rtools. #4361
• Update the AWS SDK to version 1.11.160. #4214
• Remove warnings from benchmarks. #4396
• Update clang-format to version 16. #4397
• Update the Google Cloud Storage SDK to its latest version (2.15.1). #4031
• Update capnproto to version 1.0.1. #4405
• Fix compilation with GCC 13. #4331
• Compilation fix for GCC 9.4 of Ubuntu 20.04. #4355
• Don't error out if DELETE symbol is defined. #4365

Test only changes

• Aggregates pushdown: adding tests for var size buffer overflow. #4315
• Aggregates pushdown: adding incomplete tests. #4301
• Add group backwards compatibility tests. #3996
• Aggregates: Adding tests for computation policies. #4421
• Aggregators: adding benchmark for aggregate_with_count class. #4420

Experimental Changes

• Add support for basic mimo node to TileDB task graph. #3977
• Verify that dependency-only edges using std::monostate work with nodes and edges. #3939

TileDB 2.17.4

TileDB v2.17.4 Release Notes

Defects removed

• Fix serialization of empty enumerations. #4472
• Update libwebp to version 1.3.2 to fix CVE-2023-5129. #4384

TileDB 2.18.0-rc0

TileDB v2.18.0 Release Notes

Announcements

• TileDB 2.18, targeted for release in November 2023, includes an internal release of a preview set of aggregate pushdown APIs. The APIs will be finalized in 2.19.

Disk Format

• Fix the format specification for group members. #4380
• Update fragment format spec for info on tile sizes and tile offsets. #4416

Configuration changes

• Remove vfs.file.max_parallel_ops config option. #3964

Breaking C API changes

• Behavior breaking change: tiledb_group_remove_member cannot remove named members by URI. #4391

New features

• Support custom headers on s3 requests. #4400
• Add support for Google Cloud Storage on Windows. #4138
• Query aggregates REST support. #4415
• Add random number generator global singleton. #4437

Improvements

• Fix enumeration value not found error message. #4339
• Allow forward slashes in enumeration names. #4454
• Adding extra in memory offset for var tiles on read path. #4294
• Aggregators: adding mean aggregate. #4292
• Remove parallel file I/O. #3964
• C.41 constructor: S3. #4368
• Changed the logic of the main loop in Domain::cell_order_cmp_impl<char>. #3847
• Add REST client support for delete_fragments and delete_fragments_list. #3923
• Remove the internal ValidityVector::init_bytemap method. #4071
• Use exceptions for error handling in the Metadata code. #4012
• Move min/max computation to aggregate_with_count. #4394
• Add memory tracking to Enumeration loading. #4395
• Logs for nested exceptions now incorporate all messages. #4393
• Implement read logging modes. #4386
• Move counts computation to aggregate_with_count. #4401
• Fix delete_fragments handlers according to new model. #4414
• Aggregates: change to use size instead of min cell/max cell. #4436
• Allow for empty enumerations. #4423
• Aggregates: combine sum/mean aggregators. #4440

Documentation

• Update docs for set data buffer APIs. #4381

Defects removed

• Patch curl to fix CVE-2023-38545. #4408
• Update libwebp to version 1.3.2 to fix CVE-2023-5129. #4384
• Fix missing WebP::sharpyuv installation command. #4325
• Fix re-loading enumerations from the REST server. #4335
• Save error on the context before returning error code in vfs_open. #4347
• Filter pipeline: fix coord filter type. #4351
• Fix creation of Enumerations with std::vector<bool>. #4362
• Revert change to S3 is_object, is_bucket checks. #4370
• Specify whether the HDFS backend is enabled in the string returned by tiledb_as_built_dump. #4364
• Fix fragment corruption in the GlobalOrderWriter. #4383
• Fix bug in FragmentMetadata::get_footer_size. #4212
• Global order writer: don't try to delete the current fragment if it had failed to be created. #4052
• Fix bug in ordered label reader for unsigned integers. #4404
• Add missing include object.h in group_experimental.h. #4411
• Support Azure Storage SAS tokens that don't start with a question mark. #4418
• Avoid empty Enumeration REST requests. #4369
• Make the initializer_list overload of Subarray::set_subarray zero-copy. #4062

API changes

C API

• New: Aggregate pushdown C API. #4176
• Implement tiledb_handle_load_array_schema_request. #4399
• Add ArraySchemaEvolution::extend_enumeration. #4445
• Allow null cfg in group consolidate_metadata and vacuum_metadata. #4412

C++ API

• Fix UBSAN error in the Enumeration CPP API. #4357
• Remove default operation type from query condition set membership API (introduced in 2.17). #4390
• Fix CPP set buffer APIs to work with aggregates. #4434
• Adding CPP example for consolidation plan. #4433
• New: Aggregate API for MEAN and NULL_COUNT capi/cppapi wraps. #4443

Build System Changes

• Fix static linking failures and build ExampleExe_static on Windows. #4336
• Run Windows nightly debug build on C:/ drive to avoid out of space errors. #4352
• Manage dependencies with vcpkg by default. #4348
• Find the OpenSSL dependency of the AWS SDK on non-Windows only. #4359
• Use generator expressions when setting compile options for the benchmarks. #4328
• Bump default compiler to GCC 10 and macos images from 11 to 12. #4367
• Fix and optimize the Release CI workflow. #4373
• Stop building gRPC when GCS is enabled. #4435
• Require only ucrt build from Rtools. #4361
• Update the AWS SDK to version 1.11.160. #4214
• Remove warnings from benchmarks. #4396
• Update clang-format to version 16. #4397
• Update the Google Cloud Storage SDK to its latest version (2.15.1). #4031
• Update capnproto to version 1.0.1. #4405
• Fix compilation with GCC 13. #4331
• Compilation fix for GCC 9.4 of Ubuntu 20.04. #4355
• Don't error out if DELETE symbol is defined. #4365

Test only changes

• Aggregates pushdown: adding tests for var size buffer overflow. #4315
• Aggregates pushdown: adding incomplete tests. #4301
• Add group backwards compatibility tests. #3996
• Aggregates: Adding tests for computation policies. #4421
• Aggregators: adding benchmark for aggregate_with_count class. #4420

Experimental Changes

• Add support for basic mimo node to TileDB task graph. #3977
• Verify that dependency-only edges using std::monostate work with nodes and edges. #3939

TileDB 2.17.3

TileDB v2.17.3 Release Notes

Improvements

• Allow for empty enumerations #4423

API changes

• Add ArraySchemaEvolution::extend_enumeration #4445

TileDB 2.17.2

TileDB v2.17.2 Release Notes

Defects removed

• Fix fragment corruption in the GlobalOrderWriter. #4383
• Prepend a question mark to the Azure Storage SAS token if it does not begin with one. #4426

TileDB 2.17.1

TileDB v2.17.1 Release Notes

Improvements

• Avoid empty Enumeration REST requests. #4369

Defects removed

• Revert change to S3 is_object, is_bucket checks. #4371

API changes

C++ API

• Add support for std::vector<bool> in Enumeration CPP API. #4362
• Fix UBSAN error in the Enumeration CPP API. #4357
  • This is a change from 2.17.0 in the header-only C++ API.
• Don't error out if DELETE symbol is defined. #4365

Build changes

• Compilation fix for GCC 9.4 of Ubuntu 20.04. #4355
• Fix static linking failures and build ExampleExe_static on Windows. #4336

TileDB 2.17.0

TileDB v2.17.0 Release Notes

Disk Format

• Storage format version is now 20, bringing changes to support enumerated types. (#4051).

Configuration Changes

• Add Enumeration max size configuration. #4308

New features

• Enumerated data types (aka: "categoricals" in Pandas, "factors" in R). #4051
• Support optimized creation of set membership (IN, NOT_IN) QueryConditions, replacing create+combine pattern for specifying the target set. #4164
• Add C and C++ APIs to reflect as-built (build-time) configuration options as a JSON string. #4112, #4199
• Filter pipeline support for datatype conversions based on filtered output datatype. #4165
• Add TileDB-REST support for vacuum and consolidation requests. #4229
• TileDB-REST support for dimension labels. #4084

Improvements

• Add TileDB-REST serialization for reinterpret_datatype. #4286
• Update VFS to call ListObjectsV2 from the AWS SDK, providing significant performance improvements for array opening and other listing-heavy operations. #4216
• Wrap KV::Reader::getKey calls into std::string_view. #4186
• Unify ca_file and ca_path configuration parameters. #4087
• Inline and optimize some trivial methods of the internal Array class. #4226
• Merge overlapping ranges on sparse reads. #4184
• Legacy reader: reading non written region causes segfault. #4253
• Make generic tile decompression single threaded to fix multi-query hang condition. #4256
• Add tiledb_handle_load_enumerations_request. #4288
• Add TileDB-REST array open v2 serialization support for enumerations. #4307

Defects removed

• Save error on the context before returning error code in tiledb_vfs_open #4347
• Filter pipeline: fix coord filter type. #4351
• Dense reader: fix nullable string values when reading with QueryCondition. #4174
• Fix segmentation fault in delta compressor test. #4143
• Fix VFS is_empty_bucket in C++ API. #4154
• Initialize a Query at the end of deserialization to fix QueryCondition error against TileDB-REST. #4148
• Sparse unordered w/ dups reader: adding ignored tiles. #4200
• Allow Subarray::get_est_result_size_nullable on remote arrays. #4202
• Fix version check bug when deserializing attributes. #4189
• Fix container issues in windows builds. #4177
• Unordered writes: fix deserialization for older clients. #4246
• Fix serialization of var-size property for nonempty domains. #4264
• Clear pending changes during group delete to avoid updating a deleted group. #4267
• Update filter checks for accepted datatypes. #4233
• Check if Subarray label ranges are set in Query::only_dim_label_query. #4285
• Optimize capnp traversal size usage. #4287
• Fix QueryCondition set conditions on attributes with enumerations. #4305
• Backwards compatibility: correctly handle 0 chunk variable-size tiles. #4310

API changes

C API

• Add C API attribute handle. #3982
• Add result_buffer_elements_nullable for variable size dimension labels. #4142
• Add new C APIs for dimension label REST support. #4072
• Add tiledb_as_built_dump, to reflect the build-time configuration options as a JSON string. #4199
• Add tiledb_query_condition_alloc_set_membership for optimized set membership QC creation. #4164
• Add C API handle for domain. #4053

C++ API

• Throw if incorrect type is passed to set_data_type. #4272
• Add tiledb::AsBuilt::dump class method, to reflect the build-time configuration options as a JSON string. #4199
• Add QueryCondition::create vectorized signature for set membership QC creation. #4164

Build System Changes

• TileDB core library now requires C++20. Note that the public C++ API still only requires C++17. #4034
• Fix macOS "experimental-features" build; CMake and CI QOL improvements #4297
• Libxml2 needs to be linked after Azure libraries #4281
• Adds macos experimental build to nightly CI. #4247
• Fix static linking from the GitHub Releases package on Windows. #4263
• Fix Azure build failures on non-Windows when vcpkg is disabled. #4258
• Do not install static libraries when TILEDB_INSTALL_STATIC_DEPS is not enabled. #4244
• Use tiledb:stdx stop_token and jthread; fix experimental/ build on macOS. #4240
• Export Azure::azure-storage-blobs on install when vcpkg is enabled. #4232
• Move Azure DevOps CI jobs to GitHub Actions. #4198
• Bump catch2 to 3.3.2 in vcpkg #4201
• Fix static linking to the on Windows. #4190
• Fix build with Clang 16 #4206
• Remove a workaround for a no longer applicable GCC/linker script bug. #4067
• Build curl with ZStandard support on vcpkg when serialization is enabled. #4166
• Use clipp from vcpkg if enabled. #4134
• Support acquiring libmagic from vcpkg. #4119
• Simplify linking to Win32 libraries. #4121
• curl version bump to resolve MyTile superbuild failure #4170
• Improve support for multi-config CMake generators. #4147
• Support new Azure SDK in superbuild #4153
• Update libwebp to version 1.3.1. #4211
• Fix Ninja builds on Windows #4173
• Fix azure static library linking order for superbuild #4171
• Fix vcpkg dependencies building only in Release mode. #4159
• Fix compile error on macos 13.4.1 with c++20 enabled. #4222
• Fix compilation with GCC 13 #4331

Test Changes

• Add tests for TILEDB_NOT. #4169
• Fix regression test exit status; fix unhandled throw (expected). #4311
• Fix as_built definitions in the C API compilation unit, and add validation test. #4289
• Smoke test: only recreate array when needed. #4262
• Array schema: C.41 changes, test support. #4004
• Conditionally disable WhiteboxAllocator tests to fix GCC-13 build. #4291
• Sorted reads tests: faster tests. #4278

TileDB 2.17.0-rc0

TileDB v2.17.0 Release Notes

Disk Format

• Storage format version is now 20, bringing changes to support enumerated types. (#4051).

Configuration Changes

• Add Enumeration max size configuration. #4308

New features

• Enumerated data types (aka: "categoricals" in Pandas, "factors" in R). #4051
• Support optimized creation of set membership (IN, NOT_IN) QueryConditions, replacing create+combine pattern for specifying the target set. #4164
• Add C and C++ APIs to reflect as-built (build-time) configuration options as a JSON string. #4112, #4199
• Filter pipeline support for datatype conversions based on filtered output datatype. #4165
• Add TileDB-REST support for vacuum and consolidation requests. #4229
• TileDB-REST support for dimension labels. #4084

Improvements

• Add TileDB-REST serialization for reinterpret_datatype. #4286
• Update VFS to call ListObjectsV2 from the AWS SDK, providing significant performance improvements for array opening and other listing-heavy operations. #4216
• Wrap KV::Reader::getKey calls into std::string_view. #4186
• Unify ca_file and ca_path configuration parameters. #4087
• Inline and optimize some trivial methods of the internal Array class. #4226
• Merge overlapping ranges on sparse reads. #4184
• Legacy reader: reading non written region causes segfault. #4253
• Make generic tile decompression single threaded to fix multi-query hang condition. #4256
• Add tiledb_handle_load_enumerations_request. #4288
• Add TileDB-REST array open v2 serialization support for enumerations. #4307

Defects removed

• Dense reader: fix nullable string values when reading with QueryCondition. #4174
• Fix segmentation fault in delta compressor test. #4143
• Fix VFS is_empty_bucket in C++ API. #4154
• Initialize a Query at the end of deserialization to fix QueryCondition error against TileDB-REST. #4148
• Sparse unordered w/ dups reader: adding ignored tiles. #4200
• Allow Subarray::get_est_result_size_nullable on remote arrays. #4202
• Fix version check bug when deserializing attributes. #4189
• Fix container issues in windows builds. #4177
• Unordered writes: fix deserialization for older clients. #4246
• Fix serialization of var-size property for nonempty domains. #4264
• Clear pending changes during group delete to avoid updating a deleted group. #4267
• Update filter checks for accepted datatypes. #4233
• Check if Subarray label ranges are set in Query::only_dim_label_query. #4285
• Optimize capnp traversal size usage. #4287
• Fix QueryCondition set conditions on attributes with enumerations. #4305
• Backwards compatibility: correctly handle 0 chunk variable-size tiles. #4310

API changes

C API

• Add C API attribute handle. #3982
• Add result_buffer_elements_nullable for variable size dimension labels. #4142
• Add new C APIs for dimension label REST support. #4072
• Add tiledb_as_built_dump, to reflect the build-time configuration options as a JSON string. #4199
• Add tiledb_query_condition_alloc_set_membership for optimized set membership QC creation. #4164
• Add C API handle for domain. #4053

C++ API

• Throw if incorrect type is passed to set_data_type. #4272
• Add tiledb::AsBuilt::dump class method, to reflect the build-time configuration options as a JSON string. #4199
• Add QueryCondition::create vectorized signature for set membership QC creation. #4164

Build System Changes

• TileDB core library now requires C++20. Note that the public C++ API still only requires C++17. #4034
• Fix macOS "experimental-features" build; CMake and CI QOL improvements #4297
• Libxml2 needs to be linked after Azure libraries #4281
• Adds macos experimental build to nightly CI. #4247
• Fix static linking from the GitHub Releases package on Windows. #4263
• Fix Azure build failures on non-Windows when vcpkg is disabled. #4258
• Do not install static libraries when TILEDB_INSTALL_STATIC_DEPS is not enabled. #4244
• Use tiledb:stdx stop_token and jthread; fix experimental/ build on macOS. #4240
• Export Azure::azure-storage-blobs on install when vcpkg is enabled. #4232
• Move Azure DevOps CI jobs to GitHub Actions. #4198
• Bump catch2 to 3.3.2 in vcpkg #4201
• Fix static linking to the on Windows. #4190
• Fix build with Clang 16 #4206
• Remove a workaround for a no longer applicable GCC/linker script bug. #4067
• Build curl with ZStandard support on vcpkg when serialization is enabled. #4166
• Use clipp from vcpkg if enabled. #4134
• Support acquiring libmagic from vcpkg. #4119
• Simplify linking to Win32 libraries. #4121
• curl version bump to resolve MyTile superbuild failure #4170
• Improve support for multi-config CMake generators. #4147
• Support new Azure SDK in superbuild #4153
• Update libwebp to version 1.3.1. #4211
• Fix Ninja builds on Windows #4173
• Fix azure static library linking order for superbuild #4171
• Fix vcpkg dependencies building only in Release mode. #4159
• Fix compile error on macos 13.4.1 with c++20 enabled. #4222

Test Changes

• Add tests for TILEDB_NOT. #4169
• Fix regression test exit status; fix unhandled throw (expected). #4311
• Fix as_built definitions in the C API compilation unit, and add validation test. #4289
• Smoke test: only recreate array when needed. #4262
• Array schema: C.41 changes, test support. #4004
• Conditionally disable WhiteboxAllocator tests to fix GCC-13 build. #4291
• Sorted reads tests: faster tests. #4278