TileDB v2.7.0 Release Notes

Disk Format

• Removed file __lock.tdb from the array folder and updated the format spec. Also removed config vfs.file.enable_filelocks. #2692

New features

• Publish Subarray access/functionality for use outside of core. #2214
• Add TILEDB_BYTE datatype #2721
• Expose array schema libtiledb version information #2863

Improvements

• Convert loose files thread_pool.* into a unit #2520
• Bump cmake_minimum_required to 3.21 #2532
• dynamic_memory unit, including allocator #2542
• implement windows CI functionality with github actions #2498
• implement windows crash dump file processing for windows #2657
• Added object libraries: baseline, buffer, thread_pool #2629
• ubuntu core dump processing (GA) CI #2642
• Avoid copy and set of config in tiledb_query_add_range as a performance optimization from the new subarray APIs #2740
• core file stack backtracing and artifact uploading for mac CI builds #2632
• Move Status::*Error to Status_*Error #2610
• ZStd Compressor: allocate one context per thread and re-use. #2701
• Include offset index in oversize error message #2757
• Print resource pool contained type on error #2757
• Using RETURN_NOT_OK_TUPLE in attribute.cc. #2787
• Add timestamp range to schema evolution to avoid race conditions based on schema timestamp #2776
• Convert dimension deserialize to factory #2763
• Improve object type detection performance #2792
• Query condition: differentiate between nullptr and empty string. #2802
• patch git+git: to git+https: to avoid GH access failure #2805
• Remove examples writing sparse fragments to dense arrays. #2804
• Change the ZStd filter compression level range and add defined compression level default values #2623
• Change the ZStd filter compression level range and add defined compression level default values #2811
• changes to augment dbg output on build failures #2801
• Sparse unordered w/ dups reader: process queries until full user buffers #2816
• Support var-length CHAR QueryConditions #2814
• Sparse global order reader: merge smaller cell slabs. #2818
• Fixing build errors in query condition. #2820
• Enabling refactored dense reader by default. #2808
• Switching TUPLE macros to be variadic. #2822
• Read tiles: no buffer pre-allocation for no-opt filter. #2819
• Refactored dense reader: resetting the unsplittable flag on completion. #2832
• Refactored dense reader: code cleanup. #2833
• Fixing Subarray::crop_to_tile to crop default dimensions. #2831
• Dense reader: fixing src cell offset for different cell order. #2835
• Support storing integral cells in a chunk. #2821
• Improve error handling for Array non empty domain calls. #2845
• Remove OpenArray and refactor StorageManager and FragmentInfo #2839
• Adding min/max/sum/null count to fragment info. #2830
• Fine-tune unfiltering parallelization #2842
• Removing ch9473 comments in unit-cppapi-string-dims.cc. #2837
• Global writes: writting bad validity values when coordinates are dups. #2848
• Fixing issues in VS2019. #2853
• Writer: processing var tiles before offset tiles. #2854
• Writer: moving unordered_map::emplace outside of parallel_for. #2860
• Dense reader: removing unnecessary loop around read_attributes. #2859
• Convert deserialize function of Attribute to factory function #2566
• Convert Filter class deserialize and create to factory functions #2655
• Tile metadata test: reducing amount of spew in verbose mode. #2882
• compute_results_count_sparse_string: fixing incorect memcmp. #2892
• Sparse rindex readers: fixing query resume on TileDB cloud. #2900

Deprecations

• Deprecate TILEDB_CHAR in favor of users using TILEDB_BYTE or TILEDB_STRING_ASCII for attribute datatypes. #2742 #2797
• Deprecate TILEDB_ANY datatype #2807
• Deprecate TILEDB_STRING_USC2 and TILEDB_STRING_USC4 datatypes. #2812
• Drop support for Visual Studio 2017 compiler #2847

Bug fixes

• Better windows relative path support with '/' #2607
• Fixed dangling links in README. #2712
• Handle multiple core files for stack traces and archiving #2723
• restore lost cmake code necessary for clean build on windows with -EnableAzure #2656
• Return error for nonempty_domain access regardless of local/remote status #2766
• Fix segfault in new sparse null QueryCondition code #2794
• ReaderBase needs to load var sizes #2809
• Only initialize REST query strategies once #2836
• Logger json format output is not valid #2850
• Closing a non-opened array should be a no-op instead of error. #2889
• patch (unsupported) 3rd party azure cpp lite sdk used by TileDB to avoid memory faults #2881
• Free allocated latest array schema when on error of loading all array schemas #2907

API additions

C API

• Add tiledb_array_schema_evolution_set_timestamp_range to avoid race conditions based on schema timestamp #2776
• Deprecate TILEDB_CHAR in favor of users using TILEDB_BYTE or TILEDB_STRING_ASCII for attribute datatypes. #2742 #2797
• Add {set,get}_validity_filter_list #2798
• Deprecate TILEDB_ANY datatype #2807
• Deprecate TILEDB_STRING_USC2 and TILEDB_STRING_USC4 datatypes. #2812
• Add tiledb_array_schema_get_version for fetching array schema version #2863
• Introduce experimental tiledb_ctx_alloc_with_error to return error when context alloc fails #2905

C++ API

• Add ArraySchemaEvolution::set_timestamp_range to avoid race conditions based on schema timestamp #2776
• Deprecate TILEDB_CHAR in favor of users using TILEDB_BYTE or TILEDB_STRING_ASCII for attribute datatypes. #2742 #2797
• Add validity_filter_list set/get and missing get tests #2798
• Deprecate TILEDB_ANY datatype #2807
• Deprecate TILEDB_STRING_USC2 and TILEDB_STRING_USC4 datatypes. #2812
• Add ArraySchema::version() for fetching array schema version #2863
• Add missing cstddef include to fix compile w/ GCC 7 #2885

This is a pre-release for the upcoming TileDB 2.7.0. Full history of the change will be posted in the official release. This is release should be used only for testing and validating the upcoming release. This version is not covered by TileDB compatibility or stability guarantees.

TileDB v2.7.0-rc1 Release Notes

Disk Format

• Removed file __lock.tdb from the array folder and updated the format spec. Also removed config vfs.file.enable_filelocks. #2692

New features

• Publish Subarray access/functionality for use outside of core. #2214
• Add TILEDB_BYTE datatype #2721
• Expose array schema libtiledb version information #2863

Improvements

• Convert loose files thread_pool.* into a unit #2520
• Bump cmake_minimum_required to 3.21 #2532
• dynamic_memory unit, including allocator #2542
• implement windows CI functionality with github actions #2498
• implement windows crash dump file processing for windows #2657
• Added object libraries: baseline, buffer, thread_pool #2629
• ubuntu core dump processing (GA) CI #2642
• Avoid copy and set of config in tiledb_query_add_range as a performance optimization from the new subarray APIs #2740
• core file stack backtracing and artifact uploading for mac CI builds #2632
• Move Status::*Error to Status_*Error #2610
• ZStd Compressor: allocate one context per thread and re-use. #2701
• Include offset index in oversize error message #2757
• Print resource pool contained type on error #2757
• Using RETURN_NOT_OK_TUPLE in attribute.cc. #2787
• Add timestamp range to schema evolution to avoid race conditions based on schema timestamp #2776
• Convert dimension deserialize to factory #2763
• Improve object type detection performance #2792
• Query condition: differentiate between nullptr and empty string. #2802
• patch git+git: to git+https: to avoid GH access failure #2805
• Remove examples writing sparse fragments to dense arrays. #2804
• Change the ZStd filter compression level range and add defined compression level default values #2623
• Change the ZStd filter compression level range and add defined compression level default values #2811
• changes to augment dbg output on build failures #2801
• Sparse unordered w/ dups reader: process queries until full user buffers #2816
• Support var-length CHAR QueryConditions #2814
• Sparse global order reader: merge smaller cell slabs. #2818
• Fixing build errors in query condition. #2820
• Enabling refactored dense reader by default. #2808
• Switching TUPLE macros to be variadic. #2822
• Read tiles: no buffer pre-allocation for no-opt filter. #2819
• Refactored dense reader: resetting the unsplittable flag on completion. #2832
• Refactored dense reader: code cleanup. #2833
• Fixing Subarray::crop_to_tile to crop default dimensions. #2831
• Dense reader: fixing src cell offset for different cell order. #2835
• Support storing integral cells in a chunk. #2821
• Improve error handling for Array non empty domain calls. #2845
• Remove OpenArray and refactor StorageManager and FragmentInfo #2839
• Adding min/max/sum/null count to fragment info. #2830
• Fine-tune unfiltering parallelization #2842
• Removing ch9473 comments in unit-cppapi-string-dims.cc. #2837
• Global writes: writting bad validity values when coordinates are dups. #2848
• Fixing issues in VS2019. #2853
• Writer: processing var tiles before offset tiles. #2854
• Writer: moving unordered_map::emplace outside of parallel_for. #2860
• Dense reader: removing unnecessary loop around read_attributes. #2859
• Convert deserialize function of Attribute to factory function #2566
• Convert Filter class deserialize and create to factory functions #2655
• Tile metadata test: reducing amount of spew in verbose mode. #2882
• compute_results_count_sparse_string: fixing incorect memcmp. #2892
• Sparse rindex readers: fixing query resume on TileDB cloud. #2900

Deprecations

• Deprecate TILEDB_CHAR in favor of users using TILEDB_BYTE or TILEDB_STRING_ASCII for attribute datatypes. #2742 #2797
• Deprecate TILEDB_ANY datatype #2807
• Deprecate TILEDB_STRING_USC2 and TILEDB_STRING_USC4 datatypes. #2812
• Drop support for Visual Studio 2017 compiler #2847

Bug fixes

• Better windows relative path support with '/' #2607
• Fixed dangling links in README. #2712
• Handle multiple core files for stack traces and archiving #2723
• restore lost cmake code necessary for clean build on windows with -EnableAzure #2656
• Return error for nonempty_domain access regardless of local/remote status #2766
• Fix segfault in new sparse null QueryCondition code #2794
• ReaderBase needs to load var sizes #2809
• Only initialize REST query strategies once #2836
• Logger json format output is not valid #2850
• Closing a non-opened array should be a no-op instead of error. #2889
• patch (unsupported) 3rd party azure cpp lite sdk used by TileDB to avoid memory faults #2881
• Free allocated latest array schema when on error of loading all array schemas #2907

API additions

C API

• Add tiledb_array_schema_evolution_set_timestamp_range to avoid race conditions based on schema timestamp #2776
• Deprecate TILEDB_CHAR in favor of users using TILEDB_BYTE or TILEDB_STRING_ASCII for attribute datatypes. #2742 #2797
• Add {set,get}_validity_filter_list #2798
• Deprecate TILEDB_ANY datatype #2807
• Deprecate TILEDB_STRING_USC2 and TILEDB_STRING_USC4 datatypes. #2812
• Add tiledb_array_schema_get_version for fetching array schema version #2863
• Introduce experimental tiledb_ctx_alloc_with_error to return error when context alloc fails #2905

C++ API

• Add ArraySchemaEvolution::set_timestamp_range to avoid race conditions based on schema timestamp #2776
• Deprecate TILEDB_CHAR in favor of users using TILEDB_BYTE or TILEDB_STRING_ASCII for attribute datatypes. #2742 #2797
• Add validity_filter_list set/get and missing get tests #2798
• Deprecate TILEDB_ANY datatype #2807
• Deprecate TILEDB_STRING_USC2 and TILEDB_STRING_USC4 datatypes. #2812
• Add ArraySchema::version() for fetching array schema version #2863
• Add missing cstddef include to fix compile w/ GCC 7 #2885

TileDB v2.6.3 Release Notes

Bug fixes

• compute_results_count_sparse_string: fixing incorect memcmp. #2892
• Sparse rindex readers: fixing query resume with remote arrays on TileDB cloud. #2900

This is a pre-release for the upcoming TileDB 2.7.0. Full history of the change will be posted in the official release. This is release should be used only for testing and validating the upcoming release. This version is not covered by TileDB compatibility or stability guarantees.

TileDB v2.7.0-rc0 Release Notes

Disk Format

• Removed file __lock.tdb from the array folder and updated the format spec. Also removed config vfs.file.enable_filelocks. #2692

New features

• Publish Subarray access/functionality for use outside of core. #2214
• Add TILEDB_BYTE datatype #2721
• Expose array schema libtiledb version information #2863

Improvements

• Convert loose files thread_pool.* into a unit #2520
• Bump cmake_minimum_required to 3.21 #2532
• dynamic_memory unit, including allocator #2542
• implement windows CI functionality with github actions #2498
• implement windows crash dump file processing for windows #2657
• Added object libraries: baseline, buffer, thread_pool #2629
• ubuntu core dump processing (GA) CI #2642
• Avoid copy and set of config in tiledb_query_add_range as a performance optimization from the new subarray APIs #2740
• core file stack backtracing and artifact uploading for mac CI builds #2632
• Move Status::*Error to Status_*Error #2610
• ZStd Compressor: allocate one context per thread and re-use. #2701
• Include offset index in oversize error message #2757
• Print resource pool contained type on error #2757
• Using RETURN_NOT_OK_TUPLE in attribute.cc. #2787
• Add timestamp range to schema evolution to avoid race conditions based on schema timestamp #2776
• Convert dimension deserialize to factory #2763
• Improve object type detection performance #2792
• Query condition: differentiate between nullptr and empty string. #2802
• patch git+git: to git+https: to avoid GH access failure #2805
• Remove examples writing sparse fragments to dense arrays. #2804
• Change the ZStd filter compression level range and add defined compression level default values #2623
• Change the ZStd filter compression level range and add defined compression level default values #2811
• changes to augment dbg output on build failures #2801
• Sparse unordered w/ dups reader: process queries until full user buffers #2816
• Support var-length CHAR QueryConditions #2814
• Sparse global order reader: merge smaller cell slabs. #2818
• Fixing build errors in query condition. #2820
• Enabling refactored dense reader by default. #2808
• Switching TUPLE macros to be variadic. #2822
• Read tiles: no buffer pre-allocation for no-opt filter. #2819
• Refactored dense reader: resetting the unsplittable flag on completion. #2832
• Refactored dense reader: code cleanup. #2833
• Fixing Subarray::crop_to_tile to crop default dimensions. #2831
• Dense reader: fixing src cell offset for different cell order. #2835
• Support storing integral cells in a chunk. #2821
• Improve error handling for Array non empty domain calls. #2845
• Remove OpenArray and refactor StorageManager and FragmentInfo #2839
• Adding min/max/sum/null count to fragment info. #2830
• Fine-tune unfiltering parallelization #2842
• Removing ch9473 comments in unit-cppapi-string-dims.cc. #2837
• Global writes: writting bad validity values when coordinates are dups. #2848
• Fixing issues in VS2019. #2853
• Writer: processing var tiles before offset tiles. #2854
• Writer: moving unordered_map::emplace outside of parallel_for. #2860
• Dense reader: removing unnecessary loop around read_attributes. #2859
• Convert deserialize function of Attribute to factory function #2566
• Convert Filter class deserialize and create to factory functions #2655

Deprecations

• Deprecate TILEDB_CHAR in favor of users using TILEDB_BYTE or TILEDB_STRING_ASCII for attribute datatypes. #2742 #2797
• Deprecate TILEDB_ANY datatype #2807
• Deprecate TILEDB_STRING_USC2 and TILEDB_STRING_USC4 datatypes. #2812
• Drop support for Visual Studio 2017 compiler #2847

Bug fixes

• Better windows relative path support with '/' #2607
• Fixed dangling links in README. #2712
• Handle multiple core files for stack traces and archiving #2723
• restore lost cmake code necessary for clean build on windows with -EnableAzure #2656
• Return error for nonempty_domain access regardless of local/remote status #2766
• Fix segfault in new sparse null QueryCondition code #2794
• ReaderBase needs to load var sizes #2809
• Only initialize REST query strategies once #2836
• Logger json format output is not valid #2850

API additions

C API

• Add tiledb_array_schema_evolution_set_timestamp_range to avoid race conditions based on schema timestamp #2776
• Deprecate TILEDB_CHAR in favor of users using TILEDB_BYTE or TILEDB_STRING_ASCII for attribute datatypes. #2742 #2797
• Add {set,get}_validity_filter_list #2798
• Deprecate TILEDB_ANY datatype #2807
• Deprecate TILEDB_STRING_USC2 and TILEDB_STRING_USC4 datatypes. #2812
• Add tiledb_array_schema_get_version for fetching array schema version #2863

C++ API

• Add ArraySchemaEvolution::set_timestamp_range to avoid race conditions based on schema timestamp #2776
• Deprecate TILEDB_CHAR in favor of users using TILEDB_BYTE or TILEDB_STRING_ASCII for attribute datatypes. #2742 #2797
• Add validity_filter_list set/get and missing get tests #2798
• Deprecate TILEDB_ANY datatype #2807
• Deprecate TILEDB_STRING_USC2 and TILEDB_STRING_USC4 datatypes. #2812
• Add ArraySchema::version() for fetching array schema version #2863

TileDB v2.6.2 Release Notes

Bug fixes

• Only initialize REST query strategies once #2836
• Sparse unordered w dups reader: fixing max pos calculation in tile copy. #2840

TileDB v2.6.1 Release Notes

Bug fixes

• Sparse unordered w/ dups reader: off by one error in query continuation. #2815
• Sparse unordered w dups reader: fixing query continuation with subarray. #2824

API Changes

C++ API

• tiledb::Array destructor no longer calls ::close for non-owned C ptr #2823

TileDB v2.5.4 Release Notes

Bug fixes

• Sparse unordered w/ duplicates: query returns completed when incomplete. #2825

TileDB v2.6.0 Release Notes

Improvements

• Sparse unordered with dups reader: removing result cell slabs. #2606
• Use as-installed path for TileDBConfig CMake static library imports #2669
• Check error message variable for nullptr before further use #2634
• Fixing str_coord_intersects to use std::basic_string_view. #2654
• Sparse unordered with duplicates reader: cell num fix. #2636
• Sparse unordered with dups reader: fixing initial bound calculation. #2638
• Read_tiles parallelization improvements. #2644
• Sparse global order reader: memory management unit tests. #2645
• Reduce scope of open_array_for_reads_mtx_ locks #2681
• Sparse refactored readers: better parallelization for tile bitmaps. #2643
• ZStd compressor: allocate one context per thread and re-use. #2691
• Sparse refactored readers: disable filtered buffer tile cache. #2651
• Moving coord_string from returning a std::string to std::basic_string_view. #2704
• Sparse unordered w/ dups reader: tracking cell progress. #2668
• Sparse unordered w/ dups reader: fixing var size overflow adjustment. #2713
• Enable memfs tests that were disabled by mistake #2648
• Add helpful details to memory limit error strings. #2729
• Sparse refactored readers: Better vectorization for tile bitmaps calculations. #2711
• Sort ranges for unordered with duplicate reader and exit comparisons early #2736
• Sparse refactored readers: better vectorization for query condition. #2737
• Use correct frag index in tiles creation for compute_result_space_tiles. #2741
• Use a single uint64 for cell counts #2749
• Array Schema name should be included with cap'n proto serialization #2696
• Add and use blocking resource pool #2735
• Making the allocation part of read_tiles single threaded. #2753
• Sparse unordered w/ dups reader: remove invalid assert. #2778
• Read tiles: fixing preallocation size for var and validity buffers. #2781
• Sparse unordered w/ dups: var buffer overflow on tile continuation fix. #2777
• Determine non overlapping ranges automatically. #2780
• Improve object type detection performance #2792
• patch git+git: to git+https: to avoid GH access failure #2805

Deprecations

• eliminate usage of std::iterator due to c++17 deprecation #2675

Bug fixes

• upgrade to blosc 1.21.0 from 1.14.x #2422
• Guard ZStd resource pool to fix initialization race #2699
• [C API] Add missing save_error calls in vfs_ls #2714
• Use fragment array schema for applying query condition to account for schema evolution #2698
• Don't try to read config from uninitialize storage manager #2771
• Fix segfault in new sparse null QueryCondition code #2794
• ReaderBase needs to load var sizes #2809

API additions

C API

• Add bulk point-range setter tiledb_query_add_point_ranges #2765
• Add experimental query status details API #2770
• Add {set,get}_validity_filter_list #2798

C++ API

• Backport Query::ctx and Query::array getters from 2.7 #2754
• Add validity_filter_list set/get and missing get tests #2798

This is a pre-release for the upcoming TileDB 2.6.0. Full history of the change will be posted in the official release. This is release should be used only for testing and validating the upcoming release. This version is not covered by TileDB compatibility or stability guarantees.

TileDB v2.6.0-rc2 Release Notes

Improvements

• Sparse unordered with dups reader: removing result cell slabs. #2606
• Use as-installed path for TileDBConfig CMake static library imports #2669
• Check error message variable for nullptr before further use #2634
• Fixing str_coord_intersects to use std::basic_string_view. #2654
• Sparse unordered with duplicates reader: cell num fix. #2636
• Sparse unordered with dups reader: fixing initial bound calculation. #2638
• Read_tiles parallelization improvements. #2644
• Sparse global order reader: memory management unit tests. #2645
• Reduce scope of open_array_for_reads_mtx_ locks #2681
• Sparse refactored readers: better parallelization for tile bitmaps. #2643
• ZStd compressor: allocate one context per thread and re-use. #2691
• Sparse refactored readers: disable filtered buffer tile cache. #2651
• Moving coord_string from returning a std::string to std::basic_string_view. #2704
• Sparse unordered w/ dups reader: tracking cell progress. #2668
• Sparse unordered w/ dups reader: fixing var size overflow adjustment. #2713
• Enable memfs tests that were disabled by mistake #2648
• Add helpful details to memory limit error strings. #2729
• Sparse refactored readers: Better vectorization for tile bitmaps calculations. #2711
• Sort ranges for unordered with duplicate reader and exit comparisons early #2736
• Sparse refactored readers: better vectorization for query condition. #2737
• Use correct frag index in tiles creation for compute_result_space_tiles. #2741
• Use a single uint64 for cell counts #2749
• Array Schema name should be included with cap'n proto serialization #2696
• Add and use blocking resource pool #2735
• Making the allocation part of read_tiles single threaded. #2753
• Sparse unordered w/ dups reader: remove invalid assert. #2778
• Read tiles: fixing preallocation size for var and validity buffers. #2781
• Sparse unordered w/ dups: var buffer overflow on tile continuation fix. #2777
• Determine non overlapping ranges automatically. #2780
• Improve object type detection performance #2792
• patch git+git: to git+https: to avoid GH access failure #2805

Deprecations

• eliminate usage of std::iterator due to c++17 deprecation #2675

Bug fixes

• upgrade to blosc 1.21.0 from 1.14.x #2422
• Guard ZStd resource pool to fix initialization race #2699
• [C API] Add missing save_error calls in vfs_ls #2714
• Use fragment array schema for applying query condition to account for schema evolution #2698
• Don't try to read config from uninitialize storage manager #2771
• Fix segfault in new sparse null QueryCondition code #2794
• ReaderBase needs to load var sizes #2809

API additions

C API

• Add bulk point-range setter tiledb_query_add_point_ranges #2765
• Add experimental query status details API #2770
• Add {set,get}_validity_filter_list #2798

C++ API

• Backport Query::ctx and Query::array getters from 2.7 #2754
• Add validity_filter_list set/get and missing get tests #2798

This is a pre-release for the upcoming TileDB 2.6.0. Full history of the change will be posted in the official release. This is release should be used only for testing and validating the upcoming release. This version is not covered by TileDB compatibility or stability guarantees.

TileDB v2.6.0-rc1 Release Notes

Improvements

• Sparse unordered with dups reader: removing result cell slabs. #2606
• Use as-installed path for TileDBConfig CMake static library imports #2669
• Check error message variable for nullptr before further use #2634
• Fixing str_coord_intersects to use std::basic_string_view. #2654
• Sparse unordered with duplicates reader: cell num fix. #2636
• Sparse unordered with dups reader: fixing initial bound calculation. #2638
• Read_tiles parallelization improvements. #2644
• Sparse global order reader: memory management unit tests. #2645
• Reduce scope of open_array_for_reads_mtx_ locks #2681
• Sparse refactored readers: better parallelization for tile bitmaps. #2643
• ZStd compressor: allocate one context per thread and re-use. #2691
• Sparse refactored readers: disable filtered buffer tile cache. #2651
• Moving coord_string from returning a std::string to std::basic_string_view. #2704
• Sparse unordered w/ dups reader: tracking cell progress. #2668
• Sparse unordered w/ dups reader: fixing var size overflow adjustment. #2713
• Enable memfs tests that were disabled by mistake #2648
• Add helpful details to memory limit error strings. #2729
• Sparse refactored readers: Better vectorization for tile bitmaps calculations. #2711
• Sort ranges for unordered with duplicate reader and exit comparisons early #2736
• Sparse refactored readers: better vectorization for query condition. #2737
• Use correct frag index in tiles creation for compute_result_space_tiles. #2741
• Use a single uint64 for cell counts #2749
• Array Schema name should be included with cap'n proto serialization #2696
• Add and use blocking resource pool #2735
• Making the allocation part of read_tiles single threaded. #2753
• Sparse unordered w/ dups reader: remove invalid assert. #2778
• Read tiles: fixing preallocation size for var and validity buffers. #2781
• Sparse unordered w/ dups: var buffer overflow on tile continuation fix. #2777
• Determine non overlapping ranges automatically. #2780
• Improve object type detection performance #2792

Deprecations

• eliminate usage of std::iterator due to c++17 deprecation #2675

Bug fixes

• upgrade to blosc 1.21.0 from 1.14.x #2422
• Guard ZStd resource pool to fix initialization race #2699
• [C API] Add missing save_error calls in vfs_ls #2714
• Use fragment array schema for applying query condition to account for schema evolution #2698
• Don't try to read config from uninitialize storage manager #2771
• Fix segfault in new sparse null QueryCondition code #2794

API additions

C API

• Add bulk point-range setter tiledb_query_add_point_ranges #2765
• Add experimental query status details API #2770

C++ API

• Backport Query::ctx and Query::array getters from 2.7 #2754

This is a pre-release for the upcoming TileDB 2.6.0. Full history of the change will be posted in the official release. This is release should be used only for testing and validating the upcoming release. This version is not covered by TileDB compatibility or stability guarantees.

TileDB v2.6.0-rc0 Release Notes

Improvements

• Sparse unordered with dups reader: removing result cell slabs. #2606
• Use as-installed path for TileDBConfig CMake static library imports #2669
• Check error message variable for nullptr before further use #2634
• Fixing str_coord_intersects to use std::basic_string_view. #2654
• Sparse unordered with duplicates reader: cell num fix. #2636
• Sparse unordered with dups reader: fixing initial bound calculation. #2638
• Read_tiles parallelization improvements. #2644
• Sparse global order reader: memory management unit tests. #2645
• Reduce scope of open_array_for_reads_mtx_ locks #2681
• Sparse refactored readers: better parallelization for tile bitmaps. #2643
• ZStd compressor: allocate one context per thread and re-use. #2691
• Sparse refactored readers: disable filtered buffer tile cache. #2651
• Moving coord_string from returning a std::string to std::basic_string_view. #2704
• Sparse unordered w/ dups reader: tracking cell progress. #2668
• Sparse unordered w/ dups reader: fixing var size overflow adjustment. #2713
• Enable memfs tests that were disabled by mistake #2648
• Add helpful details to memory limit error strings. #2729
• Sparse refactored readers: Better vectorization for tile bitmaps calculations. #2711
• Sort ranges for unordered with duplicate reader and exit comparisons early #2736
• Sparse refactored readers: better vectorization for query condition. #2737
• Use correct frag index in tiles creation for compute_result_space_tiles. #2741
• Use a single uint64 for cell counts #2749
• Array Schema name should be included with cap'n proto serialization #2696
• Add and use blocking resource pool #2735
• Making the allocation part of read_tiles single threaded. #2753
• Sparse unordered w/ dups reader: remove invalid assert. #2778
• Read tiles: fixing preallocation size for var and validity buffers. #2781
• Sparse unordered w/ dups: var buffer overflow on tile continuation fix. #2777
• Determine non overlapping ranges automatically. #2780

Deprecations

• eliminate usage of std::iterator due to c++17 deprecation #2675

Bug fixes

• upgrade to blosc 1.21.0 from 1.14.x #2422
• Guard ZStd resource pool to fix initialization race #2699
• [C API] Add missing save_error calls in vfs_ls #2714
• Use fragment array schema for applying query condition to account for schema evolution #2698
• Don't try to read config from uninitialize storage manager #2771

API additions

C API

• Add bulk point-range setter tiledb_query_add_point_ranges #2765
• Add experimental query status details API #2770

C++ API

• Backport Query::ctx and Query::array getters from 2.7 #2754

TileDB v2.5.3 Release Notes

Improvements

• Removing unnecessary openssl callback function. #2705
• openssl3 md5 deprecation mitigation #2716
• Sparse refactored reader: change all_tiles_loaded_ to vector of uint8_t. #2724

Bug fixes

• Properly check and use legacy readers instead of refactored in serialized query. #2667
• Set array URI in cap'n proto object for compatibility with repeated opened array usage in TileDB 2.4 and older. #2676
• Add the compute_mbr_var_func_pointer assignment in Dimension constructor #2730

TileDB v2.5.2 Release Notes

Improvements

• Provide non-AVX2 build artifact on Linux #2649
• Error out when setting multiple ranges for global layout #2658

Bug fixes

• Patch AWS sdk for cmake 3.22 support #2639
• Remove assert on memory_used_result_tile_ranges_ in SparseUnorderedWithDupsReader #2652
• Remove tiles that are empty through being filtered with a query condition #2659
• Always load array schemas during array open to find any new array schemas created from array schema evolution #2613

note: This release fixes an issue in 2.4.3 release artifacts in which the linux artifact has the library in lib but the cmake target as set to lib64

TileDB v2.4.4 Release Notes

Improvements

• use ROW_MAJOR read paths for unordered reads of Hilbert layout array #2551

Bug fixes

• Fix bug in Arrow schema construction #2554

TileDB v2.5.1 Release Notes

New features

• Disable AVX2 for MSys2 builds used by CRAN #2614

Improvements

• Clarify error messages in check_buffers_correctness() #2580

Bug fixes

• Fix schema evolution calls on all pre-TileDB 2.4 arrays #2611
• Unordered reads should be allowed for dense arrays #2608
• Fix logger creation on context to be threadsafe #2625

API additions

C++ API

• Add C++ API for Context::last_error() #2609

TileDB v2.5.0 Release Notes

This release contains a major update to the sparse unordered with duplicates reader. In many cases this is a large performance increase for arrays and query layout that operate with this. The old reader is still available and can be toggled on with the config options highlighted below.

Configuration Changes

The config parameter for sm.use_refactored_readers has been split into three, sm.query.sparse_global_order.reader, sm.query.sparse_unordered_with_dups.reader and sm.query.dense.reader which take values of refactored or legacy. #2569

Breaking C API changes

• Remove deprecated c-api tiledb_array_max_buffer_size and tiledb_array_max_buffer_size_var #2579

Breaking C++ API changes

• Remove deprecated cpp-api Array::max_buffer_elements #2579

New features

• Support upgrading an older version array to the latest version #2513
• Add improved logging support to classes #2565

Improvements

• Replace Buffer key_ with char key_[32] per shortcut story id 9561 #2502
• Remove support for sparse writes in dense arrays. #2504
• Initial dense refactor. #2503
• More concise cmake output during build #2512
• Sparse refactored readers: fixing looping behavior on large arrays. #2530
• Use sparse global order reader for unordered without duplicates queries. #2526
• Add CMakeUserPresets.json to .gitignore #2534
• Sparse unordered with duplicates reader: support multiple ranges. #2537
• Refactored sparse readers: tile overlap refactor. #2547
• Refactored dense reader: fixing output buffer offsets with multi-ranges. #2553
• Refactored sparse readers: serialization fixes. #2558
• Refactored sparse readers: proper lifetime for tile bitmaps. #2563
• REST scratch buffer is now owned by the query to allow reuse #2555
• Remove default constructor from Dimension #2561
• Resource pool: fixing off by one error. #2567
• Splitting config for refactored readers. #2569
• Sparse refactored readers: memory management unit tests. #2568
• Removed all aspects of posix_code from Status #2571
• Fixing pre-loading for tile offsets in various readers. #2570
• Use the new logger in Subarray, SubarrayPartitioner and Consolidator classes. #2574
• Add tiledb_fragment_info_get_schema_name #2581
• Enable CMake AVX2 check #2591
• Adding logging for sparse refactored readers. #2575
• use ROW_MAJOR read paths for unordered reads of Hilbert layout array #2551

Bug fixes

• Fix the memory leak in store_array_schema in the StorageManager class. #2480
• Fix curl/REST query scratch size to reset after each query is processed. #2535
• Sparse refactored readers: segfault with dimension only reads. #2539
• REST array metadata writes should post with timestamps #2545
• Fix bug in Arrow schema construction #2554
• Replaced auto& path with auto path #2560

API additions

C API

• Expose MBR in Fragment Info API #2222
• Add tiledb_fragment_info_get_array_schema_name for fetching array name used by fragment #2581

C++ API

• Expose MBR in Fragment Info API #2222
• Add FragmentInfo::array_schema_name for fetching array name used by fragment #2581

This is a pre-release for the upcoming TileDB 2.5.0. Full history of the change will be posted in the official release. This is release should be used only for testing and validating the upcoming release. This version is not covered by TileDB compatibility or stability guarantees.

Current changes include:

TileDB v2.5.0-rc1 Release Notes

Breaking Behavior

The config parameter for sm.use_refactored_readers has been split into three, sm.query.sparse_global_order.reader, sm.query.sparse_unordered_with_dups.reader and sm.query.dense.reader which take values of refactored or legacy. #2569

Breaking C API changes

• Remove deprecated c-api tiledb_array_max_buffer_size and tiledb_array_max_buffer_size_var #2579

Breaking C++ API changes

• Remove deprecated cpp-api Array::max_buffer_elements #2579

New features

• Support upgrading an older version array to the latest version #2513
• Add improved logging support to classes #2565

Improvements

• Replace Buffer key_ with char key_[32] per shortcut story id 9561 #2502
• Remove support for sparse writes in dense arrays. #2504
• Initial dense refactor. #2503
• More concise cmake output during build #2512
• Sparse refactored readers: fixing looping behavior on large arrays. #2530
• Use sparse global order reader for unordered without duplicates queries. #2526
• Add CMakeUserPresets.json to .gitignore #2534
• Sparse unordered with duplicates reader: support multiple ranges. #2537
• Refactored sparse readers: tile overlap refactor. #2547
• Refactored dense reader: fixing output buffer offsets with multi-ranges. #2553
• Refactored sparse readers: serialization fixes. #2558
• Refactored sparse readers: proper lifetime for tile bitmaps. #2563
• REST scratch buffer is now owned by the query to allow reuse #2555
• Remove default constructor from Dimension #2561
• Resource pool: fixing off by one error. #2567
• Splitting config for refactored readers. #2569
• Sparse refactored readers: memory management unit tests. #2568
• Removed all aspects of posix_code from Status #2571
• Fixing pre-loading for tile offsets in various readers. #2570
• Use the new logger in Subarray, SubarrayPartitioner and Consolidator classes. #2574
• Add tiledb_fragment_info_get_schema_name #2581
• Enable CMake AVX2 check #2591
• Adding logging for sparse refactored readers. #2575
• use ROW_MAJOR read paths for unordered reads of Hilbert layout array #2551

Bug fixes

• Fix the memory leak in store_array_schema in the StorageManager class. #2480
• Fix curl/REST query scratch size to reset after each query is processed. #2535
• Sparse refactored readers: segfault with dimension only reads. #2539
• REST array metadata writes should post with timestamps #2545
• Fix bug in Arrow schema construction #2554
• Replaced auto& path with auto path #2560

API additions

C API

• Expose MBR in Fragment Info API #2222
• Add tiledb_fragment_info_get_array_schema_name for fetching array name used by fragment #2581

C++ API

• Expose MBR in Fragment Info API #2222
• Add FragmentInfo::array_schema_name for fetching array name used by fragment #2581

This is a pre-release for the upcoming TileDB 2.5.0. Full history of the change will be posted in the official release. This is release should be used only for testing and validating the upcoming release. This version is not covered by TileDB compatibility or stability guarantees.

Current changes include:

TileDB v2.5.0-rc0 Release Notes

New features

• Support upgrading an older version array to the latest version #2513
• Add improved logging support to classes #2565

Improvements

• Replace Buffer key_ with char key_[32] per shortcut story id 9561 #2502
• Remove support for sparse writes in dense arrays. #2504
• Initial dense refactor. #2503
• More concise cmake output during build #2512
• Sparse refactored readers: fixing looping behavior on large arrays. #2530
• Use sparse global order reader for unordered without duplicates queries. #2526
• Add CMakeUserPresets.json to .gitignore #2534
• Sparse unordered with duplicates reader: support multiple ranges. #2537
• Refactored sparse readers: tile overlap refactor. #2547
• Refactored dense reader: fixing output buffer offsets with multi-ranges. #2553
• Refactored sparse readers: serialization fixes. #2558
• Refactored sparse readers: proper lifetime for tile bitmaps. #2563
• REST scratch buffer is now owned by the query to allow reuse #2555
• Remove default constructor from Dimension #2561
• Resource pool: fixing off by one error. #2567
• Splitting config for refactored readers. #2569
• Sparse refactored readers: memory management unit tests. #2568
• Removed all aspects of posix_code from Status #2571
• Fixing pre-loading for tile offsets in various readers. #2570
• Use the new logger in Subarray, SubarrayPartitioner and Consolidator classes. #2574

Bug fixes

• Fix the memory leak in store_array_schema in the StorageManager class. #2480
• Fix curl/REST query scratch size to reset after each query is processed. #2535
• Sparse refactored readers: segfault with dimension only reads. #2539
• REST array metadata writes should post with timestamps #2545
• Fix bug in Arrow schema construction #2554
• Replaced auto& path with auto path #2560

API additions

C API

• Expose MBR in Fragment Info API #2222

C++ API

• Expose MBR in Fragment Info API #2222

TileDB v2.4.3 Release Notes

Bug fixes

• Fix segfault in result ResultTile::coord_string and ResultTile::compute_results_sparse<char> due empty chunk buffer #2531
• Fix memory corruption with empty result set in extra_element mode #2540
• REST array metadata writes should post with timestamps #2545
• Backport fixes for new Sparse Unordered with Duplicate readers from #2530 and #2538

Please note that v2.4.1 was skipped due to accidental tagging of the 2.4.1 git tag during CI testing.

TileDB v2.4.2 Release Notes

New features

• Add support for empty string as query condition value. #2507

Improvements

• Support writing empty strings for dimensions #2501
• Refactored readers can segfault when multiple contexts are used. #2525

Bug fixes

• Fix ch10191: check cell_val_num for varlen status instead of result count #2505
• Do not access variables after moving them #2522
• Add try/catch to tiledb_ctx_alloc for exception safety #2527