v0.10.2 Bugfix Release

This is a bug fix release for various issues discovered after we released 0.10.1. There are no new features, just bug fixes. Database files created by DuckDB v0.10.* or v0.9.* can be read by DuckDB v0.10.2.

SQL Modifications

This release has a number of bug fixes that change SQL semantics in a few edge cases:

• Nested Boolean Comparisons now have consistent NULL comparison semantics - #11496
• Structs with non-matching keys require explicit casts when compared or combined - #11396

What's Changed

• Bump julia version & fix release script for sub-versions > 9 by @Mytherin in #11225
• Flatten Rewrite by @maiadegraaf in #11223
• ORDER BY ColumnNumber with Collations by @tiagokepe in #11139
• Fix differences to implementation for to_parquet, write_parquet, to_csv, write_csv, Expression.alias, DuckDBPyRelation.map by @binste in #11135
• Issue template: Ask for MWEs by @szarnyasg in #11192
• Cleaning up FSST: Remove unused AVX512 code by @hannes in #11222
• Fix #11211 - correctly fill in string_t padding for bit type by @Mytherin in #11231
• Fix #3391: Stop creating background threads if the thread constructor throws an exception by @Mytherin in #11236
• R_CMD_CHECK: Pin to duckdb/duckdb-r 0ed106a71c by @carlopi in #11245
• Add support for HEX(BLOB) by @Mytherin in #11243
• Remove no_vector_verification in Map Subscript Test by @maiadegraaf in #11242
• Update logos in README by @szarnyasg in #11256
• Ignore user defined parameters that change names or types of csv columns in sniffer's prompt. by @pdet in #11257
• [Python] Fix error caused by looking up a TypeCatalogEntry without an active transaction. by @Tishj in #11255
• [Fix] Fuzzer issue in list_select by @taniabogatsch in #11248
• [Parquet] Support for LZ4 Compression by @hannes in #11220
• Fix #11254: Add missing includes to terminal by @Mytherin in #11265
• Issue #10867: AsOf Predicate Pushdown by @hawkfish in #11233
• Fix plan cost runner regression script by @Tmonster in #11129
• Check if we need to throw any remaining errors at end of CSV scanning by @pdet in #11276
• Allow duplicate names in json objects when ignore_errors is true by @lnkuiper in #11271
• Do not surround JSON with quotes in sqlite shell output by @lnkuiper in #11268
• add TRIM support to virtual filesystem, and implementation on linux by @jkub in #11258
• Perform direct write operation if input data are larger than buffer size by @quentingodeau in #11203
• Fuzzer fixes by @lnkuiper in #11286
• Compile spatial also for rtools by @carlopi in #11291
• allow injecting custom BufferManager implementation by @jkub in #11215
• Default to RECORDS in JSON reader if more than one column is specified by @lnkuiper in #11295
• Add support for materialized CTEs in INSERT/UPDATE/DELETE statements by @kryonix in #10878
• Only throw exception if je_mallctl fails in DEBUG mode by @lnkuiper in #11303
• Fixing casting issue in generators by @hannes in #11304
• Rework FileSystem::OpenFile call, and add FILE_FLAGS_NULL_IF_NOT_EXISTS by @Mytherin in #11297
• Fix potential UB when list() aggregate is used in combination with other arena using aggregate functions by @Maxxen in #11306
• Fix #11293 - for ARRAY([subquery]) explicitly push the ORDER BY of the underlying subquery into the array aggregate by @Mytherin in #11316
• Fix #11281: explicitly select column types of information_schema tables for all columns, even if they are always NULL by @Mytherin in #11317
• Fixup py upload by @carlopi in #11308
• Issue #11279: TIMESTAMP => TIMESTAMPTZ by @hawkfish in #11320
• Fix null pointer exception when rolling back updates if the rollback was caused by an OOM by @Mytherin in #11309
• Fix #11283 - report consistent foreign key constraint name in information_schema by @Mytherin in #11318
• Fix #11294 - avoid applying Filter Pushdown optimization for UNION/EXCEPT without ALL by @Mytherin in #11315
• Fix #10695 - handle ? prepared statement parameters correctly for POSITION(x IN y) by @Mytherin in #11314
• Windows CLI - emit UTF8 directly using SetConsoleOutputCP(CP_UTF8) if possible by @Mytherin in #11324
• Fix #11319: use modulo when computing day of the week in excel extension by @Mytherin in #11328
• [CI] Fix bash syntax in TwineUpload by @carlopi in #11333
• Fix #11284: avoid adding the same column multiple times to a primary key/unique constraint name list by @Mytherin in #11325
• In ColumnData, limit scan to the current count in the column by @Mytherin in #11329
• Issue #11269: DISTINCT Sorted Aggregates by @hawkfish in #11321
• [Attach] Fix bug causing sequences to break attaching databases. by @Tishj in #11327
• Flatten hash vector before combining list hashes by @lnkuiper in #11340
• Make sniffer more consistent when nullpadding/ignore_errors are on by @pdet in #11313
• fix(arrow): union buffer count & handle schema errors by @Mause in #11326
• fix duckdb-r script by @Tmonster in #11345
• Fix regression_test_runner.py by @carlopi in #11346
• Issue #11234: IEJoin Scan Reset by @hawkfish in #11347
• TPC-H: Use BIGINT for ID fields schema where required by the specification by @szarnyasg in #11341
• Another round of polishing staged releases by @carlopi in #11342
• CI: Remove issue labeling workflow by @szarnyasg in #11355
• RE2 upgrade to version 2023-02-01 by @hannes in #11252
• File System: Add optional_ptr<FileOpener> to various calls, and add support for attaching DuckDB files over S3 by @Mytherin in #11343
• README: Display different logo for light/dark mode by @szarnyasg in #11366
• Fix bug in duckdb_bind_blob by @pfarndt in #11368
• Fix OSX CI by @samansmink in #11379
• Enable clang-tidy on headers and fix all headers to conform to our clang-tidy rules by @Mytherin in #11376
• Add logical_type to parameters of format_pg_type by @Flogex in #11369
• Issue #10965: RESPECT IGNORE NULLS by @hawkfish in #11372
• Fix building issues in WIN32, remove unnecessary modification. by @kindred77 in #11356
• Zero-initialize aggregate states with destructors immediately after allocating by @lnkuiper in #11360
• Update README.md by @jingshi-ant in #11357
• Issue #10885: Negative Window RANGEs by @hawkfish in #11390
• Issue #11377: Invertible TIMESTAMP_XXX Casts by @hawkfish in #11392
• Update init.py To export "extract_statements" function by @oomojola in #11394
• Internal #1657: Stricter STRUCT Casts by @hawkfish in #11396
• allow set readonly on attached db by @stephaniewang526 in #11397
• Give preference to FSSPEC defined FS by @pdet in #11400
• Default serialize optional_idx, add skip_default option to json_serialize_sql() by @Maxxen in #11405
• CI: Also label PRs as 'stale' and close them when there's no activity by @szarnyasg in #11420
• fix(jdbc): 1-index getBytes() by @Mause in #11421
• Remove redundant default descriptions by @szarnyasg in #11415
• clang-tidy: enable cppcoreguidelines-pro-type-const-cast by @Mytherin in #11414
• clang-tidy: enable `cppco...

v0.10.1 Bugfix Release

This is a bug fix release for various issues discovered after we released 0.10.0. There are no new features, just bug fixes. Database files created by DuckDB v0.10.0 or v0.9.* can be read by DuckDB v0.10.1.

What's Changed

• Remove visualizer leftovers by @Y-- in #10642
• Add explicit numbering to C enums + various compilation/CI fixes by @Mytherin in #10649
• Disable print method for CSV scanner for R build by @hannes in #10650
• Fix #10548 for the DUCKDB_NO_THREADS case by @carlopi in #10654
• Allow StorageExtension to extend DuckCatalog implementation in order to integration with observability system by @bleskes in #10643
• Update storage info for v0.10.0 by @szarnyasg in #10660
• Revamp duckdb-wasm extensions CI by @carlopi in #10672
• [CI] Re-enable skipped test window-rows-overflow.test by @Tishj in #10679
• Catch: prominently display skipped tests by @Mytherin in #10669
• Update Julia to 0.10.0 by @Mytherin in #10689
• Ingestion benchmark framework by @Tmonster in #10341
• [ICU] Add casts from Timestamp_* to TimestampTZ by @Tishj in #9539
• DISTINCT ON - greatly improve performance by rewriting ordered FIRST aggregate into arg_min_null by @Mytherin in #10684
• Fix #10685 - support aliases in join clause by @Mytherin in #10691
• Use assertThrows for throwing assertions in JDBC tests by @peteraisher in #10448
• Casts: report error location in query for failed casts by @Mytherin in #10694
• Fix duckdb spelling in _extension_deploy.yml by @carlopi in #10717
• Fuzzer #1374: ARG_XXX By Decimal by @hawkfish in #10728
• [Python] Rework the python regression test script by @Tishj in #10715
• Removes static member string by @TinyTinni in #10733
• Fuzzer #1372: Order Bind Failure by @hawkfish in #10727
• Fuzzer #1380: To Weeks Overflow by @hawkfish in #10726
• Various fixes by @carlopi in #10708
• Unittest does not satify assertion on MSVC/Debug by @TinyTinni in #10738
• Fix OrderPreservationType issue of MATERIALIZED CTEs by @kryonix in #10587
• Map creation fixes and refactoring by @taniabogatsch in #10436
• Fuzzer #1383: NULL Range Arguments by @hawkfish in #10723
• Fuzzer #1382: Window Stats Overflow by @hawkfish in #10725
• Comment on view columns by @samansmink in #10710
• Union exclude by @Tmonster in #10688
• move the logic for immediate_transaction_mode to the physical operator by @peterboncz in #10739
• [C API] Small fix and more tests by @taniabogatsch in #10748
• List_slice bug fix by @maiadegraaf in #10747
• Enable azure autoload by @samansmink in #10746
• Parquet writer - reduce memory usage of order-preserving write by @Mytherin in #10756
• Refactor csv reader includes because of r path length limitations by @hannes in #10658
• Arrow String View Type by @pdet in #10481
• local_file_system.cpp: minor fix for macOS libproc code by @barracuda156 in #10758
• Make unnamed_subquery naming predictable by @Mytherin in #10765
• [Python] Fix overflow issue in PandasAnalyzer by @Tishj in #10768
• Throw when trying to consume over 128 byte decimals by @pdet in #10601
• CLI: Right-align numerics in markdown tables by @Mytherin in #10767
• Fuzzer #1399: Window NULL RANGE by @hawkfish in #10776
• [CSV Sniffer] Minor sniffer tweak to give preference to dialects that generate the least errors if ignore_errors = true by @pdet in #10777
• Add large benchmark directory by @Tmonster in #10763
• Improve UNPIVOT error messages, and allow expressions in unpivot by @Mytherin in #10773
• Add a method UUID::FromUHugeint to generate a UUID from a uhugeint_t by @Mytherin in #10771
• [CSV Reader] Add lock to buffer reset by @pdet in #10791
• [WINDOWS] Add "/bigobj" that solves compile issue during debug by @maiadegraaf in #10782
• fix: not over-call AllSecrets by @stephaniewang526 in #10807
• Update readme by @szarnyasg in #10814
• [ODBC] Rework Connect to the ODBC driver and add functionality to set all DuckDB configurations in the Connection String by @maiadegraaf in #10692
• Fix arrow conversion, map doesn't support large offset by @yiyuanliu in #10796
• [CSV Reader] Spinlock over GetLine Error + New Strategy for dialect candidates by @pdet in #10755
• Trivial fixes by @carlopi in #10816
• Fix unicode handling in underscore of LIKE operator by @Mytherin in #10821
• JDBC spurious CI failure - an exception being thrown in this test is a race condition by @Mytherin in #10825
• Benchmark runner - allow files (e.g. CSV/Parquet) to be cached using the cache command by @Mytherin in #10817
• Fix #10803 - correctly reclaim space of list indexes when columns are dropped by @Mytherin in #10822
• [Upsert] INSERT OR REPLACE fixes by @Tishj in #10789
• [Dev] Add an optional time out in seconds to run_tests_one_by_one.py by @Tishj in #10744
• Maintain names in COLUMNS(*) expression, and allow aliasing multiple columns using {column} by @Mytherin in #10774
• Disable AWS/Azure on Windows for now by @Mytherin in #10827
• [Dev] Bump memory limit on batch_memory_usage.test_slow by @carlopi in #10845
• Fix coverity apt-get by @carlopi in #10838
• minor: FixedSizeBuffer::Pin move shared_ptr rather than copying by @mapleFU in #10837
• ci: Upgrade workflows to actions/setup-python@v5 by @krlmlr in #10832
• Fuzzer #1389: ARG_XXX Decimal Casts by @hawkfish in #10742
• Contributor guide: Fix new issue link by @szarnyasg in #10836
• Changing source to src in relational_constraints query by @Dtenwolde in #10848
• Fix: correctly calculate the range of build side for perfect hash join by @gitccl in #10446
• [Python] Fix issue caused by deadlock between thread.join() and acquiring the GIL by @Tishj in #10854
• [CSV Parser] 8-Byte Skipping instead of 1-Byte when possible by @pdet in #10855
• Add components of the version to duckdb.hpp by @ahuarte47 in #10840
• [CSV Sniffer] Tweaking header detection by @pdet in #10714
• Check if directory exists before removing files in regression test runner by @Tmonster in #10859
• [Extension] Add CatalogType to the list of functions generated in extension_entries.hpp by @Tishj in #10597
• Regression test build side probe side by @Tmonster in #10585
• [Arrow] Fix issue surrounding lifetime of dictionary arrays by @Tishj in #10610
• Fix #10745 - correctly deal with empty float columns in floating point compression routines by @Mytherin in #10863
• [Extensions] Build fixes by @carlopi in #10860
• Fix MSVC linking issue with workaround by @samansmink in #10865
• Reduce memory usage & avoid spilling to disk unnecessarily for order-preserving table creation/insertion by @Mytherin in #10862
• pb/avoid GetSchema opening a transaction by @peterboncz in #10740
• Support dollar-quoted string-constants in the CLI by @Mytherin in #10879
• Shell: avoid printing "Error: " prefix if the error message already has a prefix (e.g. Binder Error:, Parser...

DuckDB 0.10.0 "Fusca"

This release of DuckDB is named "Fusca" after the Velvet Scooter native to Europe.

Note: The on-disk storage format is backwards-compatible with the 0.9 releases of DuckDB. For details, please see the release announcement blog post.

Also note that we've dropped' the "Preview" designation with this release. DuckDB has matured quite a bit since we started creating releases back in 2019, and it is no longer appropriate.

What's Changed

• feat(jdbc): support uuid param type by @Mause in #9164
• Bump ADBC to v0.7 by @pdet in #9185
• Add support for parquet key-value metadata by @Maxxen in #9126
• Default to JSON type if objects have an inconsistent structure by @lnkuiper in #9086
• Add schema parameter to read_parquet by @lnkuiper in #9123
• [Python] Add the ability to provide a list of files to read_csv by @Tishj in #8977
• Issue #7672: TIMESTAMP_XX to DATE by @hawkfish in #9279
• N-ary lambdas, and indexes as lambda parameters by @taniabogatsch in #8851
• Feature: Fixed size list nested type (ARRAY) by @Maxxen in #8983
• Fix unused warning by @Maxxen in #9318
• Internal #215: Window EXCLUDE Functionality by @hawkfish in #9220
• Add json_serialize_plan, json_serialize_sql tweaks by @Maxxen in #9230
• Add create statement support to fuzzer by @Tmonster in #9011
• Fix for issue #8108: Random() in lambda by @maiadegraaf in #9353
• Fix Lambda Serialization by @taniabogatsch in #9323
• Allow file_row_number with parquet schema option by @samansmink in #9290
• CSV - Always run sniffer by default by @pdet in #9250
• [Python Dev] Import items lazily by @Tishj in #8741
• Array fixes + make validity more lazy by @Maxxen in #9400
• Lambda performance revamp by @taniabogatsch in #9395
• [Python] Support replacement scan on connection.table(<name>) method by @Tishj in #9427
• [Dev] Fix failure in Format Check CI job by @carlopi in #9516
• Fix parquet serialization by @Maxxen in #9485
• Support gcs:// and r2:// URLs to read data from GCS and R2 by @chrisiou in #9388
• Don't reset validity target count by @Maxxen in #9565
• Merge into feature by @carlopi in #9583
• Additional list functions by @cryoEncryp in #8907
• [Python] Support Optional[...] in DuckDBPyType by @Tishj in #8658
• C-API: support streaming arrow query by @Virgiel in #8642
• Add ToString and Print method for JoinRelationSetManager and Fix JoinNode Print by @Light-City in #9040
• Removed artificial HUGEINT minimum by @nickgerrets in #9441
• Parquet Encryption by @lnkuiper in #9392
• Internal #330: Quantile Performance by @hawkfish in #9461
• CAPI: Make it possible to create enum types by @alnkesq in #8788
• 5614 database invalidated by @Tmonster in #9513
• Add support for proper scoping (catalog + schema) to custom types by @Mytherin in #9622
• Internal #576: strptime strftime infinities by @hawkfish in #9615
• ATTACH IF NOT EXISTS by @Mytherin in #9627
• Small benchmark changes by @taniabogatsch in #9638
• add option for keep_alive setting by @samansmink in #9648
• Add "getenv" function to shell which can be used to read environment variables, and allow functions to be used in SET statements/PRAGMA statements by @Mytherin in #9651
• Julia release by @Mytherin in #9670
• In set operations ORDER BY columns refer to the first set operation in SQL - so the reference is not ambiguous by @Mytherin in #9658
• Replace old logos by @szarnyasg in #9674
• Fix dbgen/dsdgen when using custom catalog and schema by @ywelsch in #9686
• [Arrow] Properly use the parent's array.offset in many places in the scan by @Tishj in #9661
• Update issue template with API/extension repositories by @szarnyasg in #9693
• [Python] Fix lossy datetime.timedelta to INTERVAL conversion by @Tishj in #9688
• Issue template: Report vulnerabilities via dedicated channel by @szarnyasg in #9711
• Fix #9601: Call correct method in duckdb_pending_execution_is_finished by @Mytherin in #9728
• Merge Feature Into Main by @Mytherin in #9726
• feat(c): add functions for determining statement/return types by @Mause in #9430
• Issue #9673: ICU DST Truncation by @hawkfish in #9705
• Deserialize header fields by @carlopi in #9687
• fix: restore support for windows network paths by @Mause in #9725
• Constant time attach path lookup and locking to ensure unique file handles by @taniabogatsch in #9671
• [ART] WAL serialization, automatic checkpointing, decoupling catalog and storage, index names by @taniabogatsch in #9339
• Fix expanding structs in queries with ORDER BY by @taniabogatsch in #9481
• Only emit batch indices valid within the current pipeline by @ywelsch in #9702
• Fix #9717: Correctly set null statistics of children of structs by @Mytherin in #9733
• Fixes to warning and rendering of bar() by @carlopi in #9734
• Don't show "blabla" as part of syntax error by @mlafeldt in #9746
• Fix #9742: correctly catch empty ROW case in UPDATE by @Mytherin in #9744
• Explicitly attach duckdb file type by @taniabogatsch in #9747
• For ATTACH - Resolve extension prefix before determining the name so "sqlite:file.db" is again correctly aliased as "file" instead of "sqlite:file" by @Mytherin in #9753
• Wasm: Add wasm_threads as a class of built extensions by @carlopi in #9735
• Remove index joins by @taniabogatsch in #9751
• Support EXCEPT ALL and INTERSECT ALL by @hannes in #9636
• Add nightly deploy script by @samansmink in #9761
• update vcpkg by @samansmink in #9759
• Close s3 filehandle on destruction by @samansmink in #9758
• add list of collations that are required to determine equality. by @Tmonster in #9757
• Apply fix for patching vcpkg in extension workflow by @samansmink in #9767
• Internal #766: SkipList Coin Toss by @hawkfish in #9772
• Support reading large decimals into doubles in the Parquet reader by @Mytherin in #9770
• ATTACH - Always run ExtractExtensionPrefix also if a name is provided by @Mytherin in #9771
• Issue #9762: Interval Fractional Seconds by @hawkfish in #9773
• Internal #716: Summarize approx_unique BIGINT by @hawkfish in #9774
• Issue #9755: TIMESTAMP_XX DOUBLE Parts by @hawkfish in #9769
• Add support for COPY FROM DATABASE statement by @Mytherin in #9765
• chore: move links duckdblabs -> duckdb by @dpprdan in #9779
• Detect FreeBSD platform by @szarnyasg in #9782
• Extention template: Enable DuckDB-Wasm extensions by @carlopi in #9356
• Version: add info on v0.9.2 by @carlopi in #9788
• [JDBC] Sync all methods from a statement that interact with a query result by @pdet in #9659
• format_bytes rework, moving from decimal multipliers to binary ones by @carlopi in #9736
• Make FORCE CHECKPOINT abort transactions of concurrently running...

0.9.2 Bugfix Release

This is a bug fix release for various issues discovered after we released 0.9.1. There are no new features, just bug fixes. Database files created by DuckDB v0.9.0 or v0.9.1 can be read by DuckDB v0.9.2 (i.e. v0.9.2 is backwards compatible with both v0.9.0 and 0.9.1 and vice versa).

What's Changed

• [Dev] Fix Wasm.yml by @carlopi in #9303
• [Dev] Fix Wasm.yml, hardcoding extension config to latest duckdb-wasm by @carlopi in #9307
• Fix R CMD check script by @Tmonster in #9326
• [Dev] Remove unused tools/wasm folder by @carlopi in #9328
• [ODBC] SQLColAttribute fix by @maiadegraaf in #9316
• Fix typo by @shreeve in #9313
• [Dev] Trigger R.yml only only if there are changes to R.yml itself by @carlopi in #9304
• Add thread limit to test by @lnkuiper in #9305
• fix typing_extensions ImportError in experimental spark api by @Mause in #9346
• HTTPFS: Move from HTTPException to base class IOException by @carlopi in #9351
• fix for System.Data.ODBC GetSchema() by @bucweat in #9336
• Removes some static global variables by @TinyTinni in #9310
• Bump Julia to 0.9.1 by @Mytherin in #9354
• [Optimizer] Fix transitive filter issue by @Tishj in #9337
• Julia - v0.9 by @Mytherin in #9359
• fix time_bucket_tz(day): don't truncate input to a day boundary in #9320
• Increase the minimum cmake version to 3.5 by @taniabogatsch in #9364
• [PyArrow] Fix bug in timestamp pushdown by @Tishj in #9377
• Satisfy GCC's LTO checks by @krlmlr in #9386
• New generate_query_graph tool. by @Tmonster in #9212
• Fix LIST aggregate prepare statement exception by @taniabogatsch in #9370
• [Python] Throw explicit error for misuse of execute by @Tishj in #9394
• update R CMD workflow to apply patches in .github/patches/duckdb-r by @Tmonster in #9412
• [UnionVerify] Properly deal with SelectionVectors by @Tishj in #9409
• [ART] Improve error message for zero bytes in BLOBs by @taniabogatsch in #9415
• Enable serialization of LogicalCopyToFile by @Maxxen in #9418
• Nits for storage and Python API comments by @szarnyasg in #9414
• Issue #9416: Windowed Peer Functions by @hawkfish in #9425
• [Python] Adjust relation.df() to output microsecond precision for DATE types. by @Tishj in #9362
• [Python] Support PEP 563 by @Tishj in #9385
• Fix #8185 - avoid infinite recursion in AddTransitiveFilters by @Mytherin in #9440
• Fix 9447: SIGSEGV when executing read_csv() query for struct with empty VARCHAR by @ryderblue in #9448
• Fuzzer #1294: Non-Constant NULL Format by @hawkfish in #9450
• GitHub Actions bot: Prune search space with '--search' by @szarnyasg in #9445
• Issue #9396: AsOf Inequality Optimisation by @hawkfish in #9449
• Add bot for the 'Needs Documentation' label by @szarnyasg in #9444
• Fix field ids in LogicalCopyToFile:Deserialize by @Maxxen in #9424
• [Export Database] Produce up-to-date query for ViewCatalogEntry by @Tishj in #9375
• include csv_rejects_table headers for amalgamation by @Maxxen in #9454
• feat(jdbc): result streaming by @Mause in #9437
• [Arrow] Add support for dictionary's in child arrays (i.e list of ENUM) by @Tishj in #9331
• Detect AppleClang in cmake, add defines for DUCKDB_MAJOR/MINOR/PATCH_VERSION by @Maxxen in #9457
• [Dev] Fix build for extension-upload-wasm to fix (avoiding manual copy) by @carlopi in #9467
• Fix #9459 - remove unused qualified name parsing by @Mytherin in #9472
• Add license to Swift client by @szarnyasg in #9446
• Issue #8937: Window Order Collation by @hawkfish in #9477
• adding System.Data.ODBC testing to Windows.yml CI. by @bucweat in #9372
• fix(jdbc): support decimal arrays by @Mause in #9489
• Needs documentation workflow: Use correct event by @szarnyasg in #9464
• Needs documentation label: Add PR number in backlink by @szarnyasg in #9491
• Fix 9399 by @lnkuiper in #9411
• Enable option to skip building duckdb shell in extension distribution CI by @Maxxen in #9483
• fix confusing variable name by @SkyFan2002 in #9503
• Add MySQL Scanner Aliases + Enable Autoloading by @Mytherin in #9505
• Fix #9498: Amalgamation to properly define DUCKDB_VERSION by @carlopi in #9500
• Issue #582: Greenland TimeZone Change by @hawkfish in #9508
• [Union] Fix issue with keyword/quoted UNION member names. by @Tishj in #9432
• Dev: Avoid adding name_extension for extensions with DONT_LINK by @carlopi in #9496
• Implement array-based JDBC ResultSet by @elefeint in #8972
• Fix #9360, fix #9466: grab a lock before creating directories to fix race condition on Windows in partitioned write by @Mytherin in #9473
• Internal #582: Remove utc_offset Dependency by @hawkfish in #9521
• [ART] Don't allow index creation in readonly mode by @taniabogatsch in #9527
• Remove redundant class by @ccfelius in #9511
• Hive partitioning: Fix preprocessing of CreateDirectories by @carlopi in #9535
• fix: check for IsRowId before accessing column name by @stephaniewang526 in #9542
• Commit drop of index memory by @taniabogatsch in #9540
• [Catalog] Fix locking issues + identify problem in MappingValue by @Tishj in #9523
• JDBC - Add append method for LocalDateTime by @Jens-H in #9435
• Use run_tests_one_by_one to fix CI by @Mytherin in #9570
• pin run-vcpkg action version by @samansmink in #9579
• Add rowsort to test_window_order_collate.test by @carlopi in #9574
• Fix exploding Delim Joins by @Tmonster in #9564
• HivePartition: Add also lock on partition_state by @carlopi in #9576
• Fix typo by @lnkuiper in #9587
• [C-API] fix duckdb_parameter_name declaration by @suketa in #9566
• Fix incorrect template specialization by @jhammer in #9529
• fix(jdbc): allow use of nested result values after close by @Mause in #9592
• Allow pausing pipeline in NextBatch call by @ywelsch in #9562
• Optimize performance of jaccard function by @ucasfl in #9560
• [Python] Add missing default values to stubs of aggregate functions. by @Tishj in #9595
• Fix symbol leakage check by @Mytherin in #9604
• Issue: #8867 Fix: remove unused variables for logical_root. by @Light-City in #8866
• CI: Add job for 'needs maintainer approval' PR tag by @szarnyasg in #8853
• Ci: Fix trigger of 'needs maintainer approval' job by @szarnyasg in #9610
• fix(jdbc): combine jar publish steps by @Mause in #9484
• Increment postgres_scanner version by @Mytherin in #9504
• Improve linear probing performance of GroupedAggregateHashTable by @lnkuiper in #9575
• Expose InterruptState in NextBatch by @ywelsch in #9606
• [Arrow] Support LargeString and LargeList by @Tishj in #9471
• Issue #9589: Prefer strict TIMESTAMPs by @hawkfish in #9600
• Fix issue with streaming query results by @samansmink in https://g...

0.9.1 Bugfix Release

This is a bug fix release for various issues discovered after we released 0.9.0. There are no new features, just bug fixes. Database files created by DuckDB v0.9.0 can be read by DuckDB v0.9.1 (i.e. v0.9.1 is backwards compatible with v0.9.0 and vice versa).

What's Changed

• Fix missing header in nightly GCC build by @Maxxen in #9109
• Add scripts for cancelling/rerunning workflows by @Mytherin in #9120
• Update R-CMD-Check workflow by @Tmonster in #9128
• CSV error message fix by @pdet in #9132
• Throw SerializationException instead of InternalException by @Maxxen in #9130
• Radix HT: Get max_threads from config instead of system by @lnkuiper in #9129
• Wrapping CSV State Machine by @pdet in #9131
• [CSV] Properly skip new lines mid csv file by @pdet in #9139
• [PythonDev] Rework duckdb_cursor fixture in pytest by @Tishj in #9140
• Update logo in README by @szarnyasg in #9147
• Bump uncovered lines by @carlopi in #9102
• Python: Uniform DuckDB SQL version to v0.X.Y by @carlopi in #9095
• [PythonDev] Don't dereference None when creating pandas dataframe by @Tishj in #9127
• fix: fix some typo by @shizuocheng in #9156
• [Swift] fix row count and index mapping by @tcldr in #9149
• Run Single-Threaded CSV Parser when only one thread is available by @pdet in #9148
• [ODBC] Add missing fields to SQLColAttribute by @maiadegraaf in #9146
• Merge main into feature by @Mause in #9163
• [Python] Pyarrow integer filter pushdown bug fix by @Tishj in #9155
• [CSV] Be sure to run header detection even if a header is defined by the user by @pdet in #9168
• Avoid (Try)AutoLoad logic if extension already loaded by @carlopi in #9177
• Fix fuzzer exclusion by @Tmonster in #9103
• [PythonDev] Fix #duckdb-internal/418 by @Tishj in #9184
• chore: skip arrow union tests on pyarrow<11 by @gforsyth in #9186
• Fix issue related to map() and Dictionary Vectors. by @Tishj in #9046
• Issue #9183: Arbitrary ASOF Conditions by @hawkfish in #9188
• Add instructions for jdk installed via brew by @nicku33 in #9192
• Fix nits in API comments by @szarnyasg in #8768
• Internal #445: Destruct All Aggregates by @hawkfish in #9202
• Don't initialize data chunk in ExpressionExecutor if it's just a BoundReferenceExpression by @lnkuiper in #9213
• [ODBC] Add info to fatal error in compiler by @maiadegraaf in #9218
• Rewrite filter remover arrow test by @lnkuiper in #9217
• [ODBC] Add Unittests to Windows CI Run by @maiadegraaf in #9216
• Add missing parameter defaults for create_function in duckdb-stubs by @earwig in #9224
• Several fixes related to vector_size=2 nightly build by @pdet in #9166
• Avoid keeping read-only transactions stored in old_transactions by @Mytherin in #9236
• Fix reusable workflow for OOTE building by @samansmink in #9229
• [Parquet] Implement reading byte stream split encoded data by @adamreeve in #9240
• Bug report template: ask for imports to be included by @szarnyasg in #9197
• Remove bundled sqlite, does not seem to be used anywhere by @hannes in #9245
• update azure extension by @samansmink in #9243
• Fix #8413: Avoid pushing collations to non-varchar columns, and cleanup around PushCollation by @Mytherin in #9242
• Fix #8624 - allow changing of schema and search_path parameters even if configuration is locked by @Mytherin in #9244
• [ODBC] SQLColAttribte: More tests by @maiadegraaf in #9200
• Fix for creating ephemeral schema on readonly by @nicku33 in #9261
• Wasm extensions Makefile changes by @carlopi in #9269
• Fix #9241 - avoid pushing filters through DISTINCT ON by @Mytherin in #9270
• Fix #7880 part 1 - rebind expression during alias replacement instead of copying the already bound expression by @Mytherin in #9274
• ICU: Remove unused variables by @carlopi in #9275
• Fix #9252 - avoid overwriting start value as part of INCREMENT BY in sequence creation by @Mytherin in #9272
• [Python] Add back value_counts by @Tishj in #9278
• [SparkAPI] Add read.json and read.parquet + some unhappy path testing by @Tishj in #9276
• [Arrow][Dev] Make each produced Array own its own memory. by @Tishj in #9254
• Properly register all JSON cast functions so the binder can disambiguate by @lnkuiper in #9209
• [Dev] Fix test_map_vector_types.test by @Tishj in #9277
• Issue #9280: Parquet TIME_TZ Reading by @hawkfish in #9283
• Update iceberg extension by @samansmink in #9284
• Fix prepare statement issue by @taniabogatsch in #9288
• [Arrow] Fix issue with scanning of UNION of STRUCTS by @Tishj in #9291
• Remove nodejs client by @hannes in #9287
• Clear cached buffers before emitting next chunk by @pdet in #9295
• [Dev] Restore _extension_client_tests.yml syntax by @carlopi in #9300
• Fix MacOS exception catching by @carlopi in #9286
• Unify replacement scan filename parsing by @lnkuiper in #9273
• Extension install fixes by @carlopi in #9294
• Re-add windows extension builds for R by @hannes in #9302

0.9.0 Preview Release "Undulata"

This preview release of DuckDB is named "Undulata" after the aptly named Yellow-billed duck native to Africa.

Note: Again, this release introduces a backwards-incompatible change to the on-disk storage format. We suggest you use the EXPORT DATABASE command with the old version followed by IMPORT DATABASE with the new version to migrate your data. See the documentation for details.

What's Changed

• [Dev] Merge master into feature by @Tishj in #7535
• Issue #7563: make_timestamptz by @hawkfish in #7597
• Add support for nested laterals by @arhamchopra in #7528
• Issue #7563: epoch_us(temporal) by @hawkfish in #7629
• Fix lingering clang-tidy issues by @Mytherin in #7670
• Add list_intersect, list_has_any, and list_has_all by @maiadegraaf in #7518
• Issue #7563: epoch_xs(temporal) by @hawkfish in #7648
• Pivot - dynamically switch between using filtered aggregates or the new pivot operator by @Mytherin in #7688
• Add wildcard to JSON Path by @lnkuiper in #7624
• [Dev] Add optional build flag to disable assertions in debug mode by @Tishj in #7618
• [DEV]: ICU C Casts by @hawkfish in #7715
• List_resize by @maiadegraaf in #7401
• Issue #7187: AsOf Join Performance by @hawkfish in #7607
• Some minor CI changes by @samansmink in #7763
• Binder coverage by @hawkfish in #7791
• Vacuum Completely Deleted Row Groups by @Mytherin in #7794
• Issue #7187: AsOf Coverage by @hawkfish in #7774
• Implement FIELD_IDS for parquet writes by @lnkuiper in #7696
• Optimize Regexp_matches to LIKE statements when possible by @Tmonster in #7264
• Jemalloc configuration, more buffer allocator, and remove redundant string copying in parquet dictionary by @lnkuiper in #7697
• Truncate Database File on Checkpoint by @Mytherin in #7824
• LEFT JOIN ON TRUE support by @taniabogatsch in #7821
• Issue #7809: Segment Tree Performance by @hawkfish in #7831
• C Data Interface: duckdb_arrow_scan and duckdb_arrow_array_scan by @angadn in #7570
• Update Julia to 0.8.1 by @Mytherin in #7932
• Add conn.interrupt() to DuckDB python API by @henrinikku in #7895
• renaming part of extension build refactor PR by @samansmink in #7926
• fix swapped x/y regression parameters by @MartinNowak in #7855
• [Docs] Aggregate function README.md by @hawkfish in #7881
• PhysicalPiecewiseMergeJoin improvement by @xuke-hat in #7832
• Initial set of commits to add support for zOS (an IBM mainframe operating system) by @v1gnesh in #7805
• test(nodejs): add test_all_types.test.ts by @Mause in #7740
• Issue #7879: Missing JDBC TIMESTAMP_TZ by @hawkfish in #7922
• Attempt to fix CI on Windows 32 and Python on Windows by @carlopi in #7961
• Fix 7947 by @lnkuiper in #7963
• test: patch test_7652 to skip on pyarrow<11 by @gforsyth in #7966
• NodeJS: Add columns() method to get type info from prepared statement by @Maxxen in #7948
• Fix: Don't free arrow children explicitly by @Maxxen in #7917
• CSV Rejects table by @Maxxen in #7681
• Issue #7809: Segment Tree Performance by @hawkfish in #7891
• Add tpch benchmark run exclusively on parquet files by @Tmonster in #7519
• Bidirectional check storage + minor CI fixes by @carlopi in #7955
• [Swift] fix #7985 by @tcldr in #7993
• Move @samansmink's extension_header_rename.patch by @carlopi in #8001
• [Python] Properly use NumPy array stride when scanning object arrays. by @Tishj in #7964
• CI - No longer run on PR synchronize, instead run on ready_for_review by @Mytherin in #8007
• Parallel pipeline execution should call NextBatch on first batch by @bleskes in #7978
• Micro-optimization for generating collation keys by @Krechals in #7983
• Multiple assignment for UPDATE SET by @nickgerrets in #7968
• CI job to move synchronized PRs to draft by @carlopi in #8010
• [ADBC] ConnectionGetTableSchema and StatementSetSubstraitPlan Functions by @pdet in #7914
• Issue #7852: Window Vectorisation by @hawkfish in #7996
• Moving JDBC Linux x64 builds to CentOS 7 by @hannes in #7991
• CI Draft - token is called GH_TOKEN by @Mytherin in #8016
• Add support for materialized CTEs by @kryonix in #7126
• Reduce memory usage of Parquet writer by @lnkuiper in #7995
• CI auto draft: pass token via environment + avoid wrapping action by @carlopi in #8024
• CI autodraft: use implicit variable [test] by @carlopi in #8027
• remove duplicate pivots declare by @douenergy in #7992
• Fix typo in fts indexing exception by @alexanderchiu in #8034
• Fix issue 7988 by @samansmink in #8023
• Delete DraftMe.yml by @Mytherin in #8048
• Fix 3eb9ab3: Remove unneeded move by @carlopi in #8038
• [CI] Skip many more CI jobs for pull requests, and add make coverage-check to run coverage locally by @Mytherin in #8046
• Extension build configuration refactor by @samansmink in #7735
• Compressed Materialization by @lnkuiper in #7644
• [Relation] Add support for creating an empty ValueRelation by @Tishj in #7967
• Join Order Optimizer has duplicate enumerations and lost some neighbors by @lokax in #7358
• Fix CI wasm by @carlopi in #8057
• [CI] More CI reduction and clean-up by @Mytherin in #8052
• Restore auto-draft functionality by @carlopi in #8058
• Move WebAssembly.yml to NightlyTests.yml by @carlopi in #8060
• Unskip, attach HTTPFS test, and create HTTPState when the opener is not available by @pdet in #8012
• CI fixes: Don't persist ccache for nightlies by @carlopi in #8075
• Fix regression & fix draft mechanism by @carlopi in #8071
• CI compliance feature branch by @carlopi in #8070
• Fix python flaky test (potentially GET requests gets back 403) by @carlopi in #8074
• [Arrow] Fix segfault in appending list data by @Tishj in #8042
• Issue #7852: Window Vectorisation by @hawkfish in #8050
• CONTRIBUTING.md by @carlopi in #8077
• Add ORDER BY clause to query in test_bool.test by @Flogex in #8082
• ART test and benchmark refactor by @taniabogatsch in #8055
• Update plan cost runner script to remove 20% threshold for cardinality estimates by @Tmonster in #7989
• Fix #8067 by @lnkuiper in #8090
• ART prefix refactor by @taniabogatsch in #7930
• Bump Substrait by @pdet in #8110
• Merge feature into master by @Mytherin in #8136
• Increase memory limit in test to prevent non-deterministic CI failures by @lnkuiper in #8138
• UNNEST binder fix by @taniabogatsch in #8111
• Out-of-Core Hash Aggregate by @lnkuiper in #7931
• Add Unittests for ODBC by @maiadegraaf in #7875
• Hive types by @lverdoes in #7674
  ...

0.8.1 Bugfix Release

This is a bug fix release for various issues discovered after we released 0.8.0. There are no new features, just bug fixes. Database files created by DuckDB v0.8.0 can be read by DuckDB v0.8.1 (i.e. v0.8.1 is backwards compatible with v0.8.0). Note that database files created by v0.8.1 cannot be read by DuckDB v0.8.0 (i.e. v0.8.0 is not forwards compatible with v0.8.1).

Changes

• [Julia] Update DuckDB_jll to v0.8.0 by @Mytherin in #7568
• CSV reader - allow parallel option to be set in COPY statement as well by @Mytherin in #7579
• shell: Remove .dbinfo command. by @omo in #7569
• Catalog::LookupEntry(): Remove unused code. by @omo in #7557
• Add the default scheme to the CREATE TYPE's type search path. by @omo in #7555
• Use std::all_of instead of raw loop in Disjoint. by @ttsugriy in #7549
• feat: introduce a common grammar/types file for libpgquery parser and update Python scripts to take source/target directory paths as argument by @stephaniewang526 in #7574
• Fix #7582 - correctly set "last_offset" in InitializeScanWithOffset and turn assertion into run-time check by @Mytherin in #7586
• Partially fix #7551 - throw internal exception in case of type mismatch in ExpressionExecutor by @Mytherin in #7587
• Fix #7602 - allow reserved keywords in named parameters by @Mytherin in #7604
• Fix #7599 - output a clear error message when a subquery is used in a table function that does not support it by @Mytherin in #7603
• Rework Code Coverage CI - Remove CodeCov and instead track uncovered lines explicitly + turn lack of coverage into a CI failure by @Mytherin in #7611
• Use unordered_set insert range overload. by @ttsugriy in #7615
• Reserve expression_costs storage. by @ttsugriy in #7608
• [ADBC] Testing Unhappy Paths, Fixing Memory Leaks from Error Setting, Removing Macros by @pdet in #7589
• Windows - path is only absolute if path starts with a single back-slash by @Mytherin in #7623
• Fix #7564 - if the auto-complete extension is not enabled, inline it into the shell by @Mytherin in #7621
• Remove 2 extra bytes from magic string pattern. by @ttsugriy in #7626
• Avoid unnecessary table lookup. by @ttsugriy in #7630
• Reserve enough storage for unbound_expressions. by @ttsugriy in #7627
• Increment code coverage by @Mytherin in #7636
• Remove all C-style casts and add clang-tidy rule to forbid them by @Mytherin in #7656
• Fix sql auto complete extension CI issue by @Mytherin in #7650
• Add missing entries to ParquetDecodeUtils::BITPACK_MASKS by @Tishj in #7658
• Fix: allow distinct and order by in list aggregates by @taniabogatsch in #7638
• Rework the AggregateExecutor interface to no longer have unnecessary pointers and arrays by @Mytherin in #7671
• Fix #7660 - avoid exporting the same catalog multiple times in EXPORT by @Mytherin in #7676
• Move BindUpdateConstraints into a virtual function that is implemented by the DuckTableEntry by @Mytherin in #7679
• Fix #7567 - when setting the schema to a different schema within another catalog, keep the correct catalog by @Mytherin in #7678
• Fix exception fmt by @carlopi in #7683
• Fix amalgamation build by avoiding overloading multiplication by @carlopi in #7661
• Fix #7659 - use correct catalog when replaying a CREATE TABLE in the WAL by @Mytherin in #7675
• Implement #7662 - add the "lock_configuration" setting which allows configurations to be locked down by @Mytherin in #7682
• Fix #7663 - add in_search_path function, correctly show temporary views in SHOW TABLES, and show views in SHOW ALL TABLES by @Mytherin in #7680
• expose the StripUnicodeSpaces parser utility method by @stephaniewang526 in #7705
• Add FuzzyDuck fuzzer - and move fuzzer CI to separate repo by @Mytherin in #7712
• Add missing std::move for old GCCs by @Mytherin in #7714
• [Dev] Fix failing assertion in python debug by @Tishj in #7722
• Fix crash in ArrowTableFunction::GetArrowLogicalType on Linux by @Tishj in #7718
• Allow core duckdb to handle unrecognized JDBC configuration by @elefeint in #7713
• [ADBC] Transactions and explicitly not-supporting Partition Reading/Execution by @pdet in #7639
• Verify that Parallel CSV Reader skips lines mid-threads by @pdet in #7637
• Fix issue with setup.py builds without dependencies by @samansmink in #7695
• [Python] Fix tests for Pandas 2.0.2 by @Tishj in #7726
• Code Coverage CI check - allow one uncovered line by @Mytherin in #7724
• Generate default_types from json files by @Tishj in #7646
• Fix fuzzer issues found by new fuzzer CI runs by @Mytherin in #7736
• [Python] Fix conversion of deeply nested dictionaries by @Tishj in #7739
• Fix TupleDataCollection List serialization by @lnkuiper in #7741
• Fuzzer #156: Copy Before Swizzle by @hawkfish in #7747
• Minor fixes to failing CI runs by @carlopi in #7768
• Fix more fuzzer issues found by new fuzzer CI by @Mytherin in #7759
• Add option to disable serialization by @stephaniewang526 in #7745
• fix(httpfs): correct listobjectv2_url for strict s3/http servers by @Mause in #7761
• Fuzzer #209: Multiple Scalar Blocks by @hawkfish in #7764
• Fuzzer #206: Fix Cast Overflow by @hawkfish in #7770
• More minor CI fixes by @Mytherin in #7779
• Add Exception on dependency verification for Enum Types and Temp Tables by @pdet in #7641
• Add fuzz_all_functions fuzzer, and add support for varargs to test_vector_types by @Mytherin in #7754
• JSON fixes by @lnkuiper in #7762
• [Julia] Fix issue related to table function callbacks and IO by @Tishj in #7783
• [Dev] Use sql in the python_regression_test.py. by @Tishj in #7787
• Allow core duckdb to handle unrecognized C API configuration by @elefeint in #7804
• Fuzzer #214: ROWS BETWEEN Overflow by @hawkfish in #7767
• Add tests to cover issue 5132 and enable force reload by @taniabogatsch in #7800
• Fuzzer #215: Timestamp Arithmetic Overflow by @hawkfish in #7769
• Remove grammar support for CREATE/DROP DATABASE by @stephaniewang526 in #7806
• Serialize: fix some uncovered cases, part 1 by @carlopi in #7810
• CodeCov tweaks by @carlopi in #7815
• fix(jdbc): arrow error handling by @Mause in #7814
• Fix duck fuzzer #218 and #220 by @carlopi in #7818
• Add msan and ubsan to cifuzz (+ fix zstd + msan) by @carlopi in #7813
• Art bug fixes by @taniabogatsch in #7801
• Check GlobalSortState for external scan in PhysicalWindow by @lnkuiper in #7827
• remove un-used PGNodeTag by @stephaniewang526 in #7833
• refactor(fsspec): remove seekable flag by @Mause in #6585
• Unnest_rewriter fixes by @taniabogatsch in #7836
• [Julia] Fix comments on #7783 by @Tishj in #7843
• Disable attaching on-disk DuckDB databases if external access is disabled by @Mytherin in #7850
• Fix #7711 - disallow detaching the currently USEd database by @Mytherin in #7851
• [Python] only execute in DuckDBPyRelation::Close if it was never executed before by @Tishj in #7844
• Add rel_from_table_function to R relational API by @hannes in https://github.com/duckdb/d...

0.8.0 Preview Release "Fulvigula"

This preview release of DuckDB is named "Fulvigula" after the Mottled duck (Anas fulvigula) which lives in the Gulf of Mexico, where it is apparently highly prized amongst (heartless) hunters.

There are two SQL-level breaking changes in this release:

• #7174 The default sort order switched from NULLS FIRST to NULLS LAST because this is more intuitive, especially in conjunction with LIMIT.
• #7082 The division operator / will now always lead to a floating point result even with integer parameters. The new operator // retains the old semantics. This change is consistent with Python.

Note: Again, this release introduces a backwards-incompatible change to the on-disk storage format. We suggest you use the EXPORT DATABASE command with the old version followed by IMPORT DATABASE with the new version to migrate your data. See the documentation for details.

What's Changed

• Issue 5984 #4 LogicalColumnIndex out of range Error by @Tmonster in #6303
• Implementing Integration with PyTorch by @pdet in #6295
• Implement #4941: Python client: for streaming fetches construct a streaming result (fetch_one, record_batch_reader, etc) by @Mytherin in #6346
• Implement sharable Buffer Pool across DatabaseInstances by @jkub in #6299
• Add table functions range and generate_series for TIMESTAMPTZ by @papparapa in #6285
• Add Initial DuckDB Swift API by @tcldr in #6351
• Integration with TensorFlow Tensors by @pdet in #6348
• Windows - remove delayload code and enable statically linking extensions by default by @Mytherin in #6399
• Add support for Pivot/Unpivot statements by @Mytherin in #6387
• [C-API] Add support for StreamQueryResult by @Tishj in #6318
• [Swift] add remaining non-composite types by @tcldr in #6422
• [Swift] Add Prepared Statements by @tcldr in #6459
• [Python] Exclude jemalloc files while pip install on Android OS by @papparapa in #6450
• CI: Swap cron for repository_dispatch by @carlopi in #6498
• CI improvements + add version badge to README by @carlopi in #6493
• Storage: store lists as uint64 offsets instead of as list_entry_t by @Mytherin in #6499
• two changes facilitating sending table/column stats over the wire (M… by @peterboncz in #6440
• Rework Value class internals to have a similar structure to LogicalType and others by @Mytherin in #6503
• Remove unswizzle flag from SortedData::Unswizzle by @lnkuiper in #6501
• [Swift] Add Appender by @tcldr in #6482
• JDBC: Remove DuckDBDatabase by @MariusVolkhart in #6426
• Add nan and inf arithmetic by @Tmonster in #6415
• Update tools/rpkg README.md by @Tishj in #6530
• Merge feature into master by @Mytherin in #6534
• Restrict threads for reliability. by @hawkfish in #6540
• Replace replace with format strings by @domoritz in #6542
• Add missing escape for " by @domoritz in #6543
• Blob <-> Bitstring casting by @LindsayWray in #6488
• Mapfunctions: map_entries, map_values, map_keys by @LindsayWray in #6522
• Issue #5920: Ordered Aggregate Buffering by @hawkfish in #6539
• Handle SQL-tagged strings correctly with dplyr::tbl, fixes #6506 by @rsund in #6536
• CI: Update Swift.yml by @carlopi in #6553
• Update SwiftRelease.yml by @carlopi in #6554
• Java: Implement JDBC 4.1 by @MariusVolkhart in #6376
• Bitstring aggregations by @LindsayWray in #6417
• Make our default threads setting Cgroup-aware on Linux by @Tishj in #6550
• [Swift] Add composite type support by @tcldr in #6557
• Statistics Rework: Switch to single BaseStatistics class, use separate static classes for methods on the stats instead by @Mytherin in #6560
• Introduce Syntax for SEMI and ANTI joins by @Tmonster in #6480
• Update storage_info with version 0.7.1 by @carlopi in #6572
• [Python] Add the ability to supply a DuckDBPyRelation instance to register by @Tishj in #6483
• [Python] map now defaults to original type when analyzed type at bind is NULL by @Tishj in #6571
• [Dev] Fix broken test_filesystem.py test by @Tishj in #6582
• CI: Node.js, add common NPM-setup step by @carlopi in #6590
• build: add builds for nodejs linux arm64 by @Mause in #6586
• CI: move to setup-node@v3 by @carlopi in #6596
• Issue #6604: TIMESTAMP <=> TIMESTAMPTZ by @hawkfish in #6605
• [Python] Add support for EXPLAIN ANALYZE to explain method by @Tishj in #6561
• Add ICU list functions generate_series and range by @papparapa in #6445
• feat(nodejs): add errorType attribute to DuckDbError by @Mause in #6434
• Fix TPC-DS date insertion by @ywelsch in #6591
• Fix #4016: Test amalgamation with --split param by @carlopi in #6587
• feat(python): throw HTTPExceptions instead of IOException for http errors by @Mause in #6533
• Add httpfs config to support packaging it as an extension by @ankrgyl in #6608
• Issue #6595: N-Ary Positional Joins by @hawkfish in #6598
• [Swift] inline documentation plus API tweaks by @tcldr in #6614
• Fix #6602: add inet extension to build/distribute script by @Mytherin in #6610
• CI remove amalgama x8 + swift release by @carlopi in #6615
• Fix too many open file handles during JSON schema detection by @lnkuiper in #6613
• Issue #6580: Parquet Int96 Timestamps by @hawkfish in #6601
• Exception_static_build defalt: Partial revert of dabbead by @carlopi in #6620
• Make DISTINCT ON respect the ORDER BY clause similar to Postgres + several ordered aggregate improvements by @Mytherin in #6616
• fix url encode issue for R2 by @samansmink in #6609
• [Swift] Database.Configuration type + documentation enhancements by @tcldr in #6617
• R: Avoid passing SEXP by reference by @krlmlr in #6475
• Test and fix preservation of class attribute in external pointers by @krlmlr in #6526
• Add support for lambda functions to COLUMNS, and allow COLUMNS to be used in the ORDER BY/WHERE clauses by @Mytherin in #6621
• [R] Remove duplicate occurrence of dependency by @Tishj in #6625
• Automatically Fully Download Files through HTTPFS if no length header is provided by @pdet in #6448
• Remove some function calls that can throw potential false positives in CI by @Tmonster in #6623
• [Python] Add __getattr__ and __getitem__ implementations for DuckDBPyRelation by @Tishj in #6624
• [Optimizer] Regex Optimization Rule fix by @Tishj in #6634
• [Bug Fix] Enum Serialization by @pdet in #6040
• Update interval for arrow by @handstuyennn in #6515
• SQLLogicTest - instead of moving prepared statements over avoid restarting database when there are prepared statements by @Mytherin in #6638
• Bind replace table function by @samansmink in #6639
• Fix #6630: correctly set bind_data->types in the Parquet scan when using union_by_name by @Mytherin in #6642
• [Python] read_csv can now read from a file-like object. by @Tishj in #6568
• Fix #6640: correctly throw an error on altering schemas by @Mytherin in #6643
• Support multiple aggregates in top-level pivot by @m...

0.7.1 Bugfix Release

This is a bug fix release for various issues discovered after we released 0.7.0. There are no new features, just bug fixes. Notably, there is no incompatibility with database files created with v0.7.0

Changes

• When building extensions we need to add _storage_init to the whitelist on MacOS by @Mytherin in #6243
• Some more read_json_auto bugfixes by @lnkuiper in #6244
• Fix for Thrift.h: std::iterator is deprecated by @hannes in #6250
• Add missing shell mode descriptions by @papparapa in #6256
• Fix #6255: Shell should be installed in INSTALL_BIN_DIR by @Mytherin in #6266
• Bump Julia to v0.7.0 by @Mytherin in #6280
• Skip headers in read_csv functions as well by @pdet in #6267
• Correctly compute Windows terminal width, and add a .maxwidth option to the shell for duckbox mode by @Mytherin in #6274
• Fix lateral join bug by @taniabogatsch in #6268
• fix: add storage_version_info entry for v0.7.0 by @Mause in #6279
• Fix to #5461 by @annnei in #6265
• CI fixes by @Mytherin in #6289
• [Fuzzer] Fixes fuzzer issue 11 by @Tishj in #6191
• Partially Fix #6253: Improve handling of timezones in the regular VARCHAR -> TIMESTAMP cast by @Mytherin in #6283
• Error message on no content-length header by @samansmink in #6293
• fixes #6238 by @rpbouman in #6239
• fixes #6236 by @rpbouman in #6252
• Missing extension exceptions by @lverdoes in #6294
• feat: allow extensions to implement CREATE/DROP DATABASE by @rjatwal in #6115
• fix(python): python object types in stubs by @Mause in #5732
• Fix UPSERT binding issue related to the source table_index by @Tishj in #6275
• fix: DESCRIBE does not show primary key by @gkaretka in #6068
• Fix #6276: avoid transforming the root arg of a case expression multiple times by @Mytherin in #6300
• More read_json(_auto) bugfixes by @lnkuiper in #6281
• JDBC: Expand Blob, add UUID support by @MariusVolkhart in #6302
• CMake: Move from GREATER_EQUAL to GREATER, fixing #5528 by @carlopi in #6310
• Implement #6003 - add names option to CSV reader by @Mytherin in #6308
• CI: Test for cron based workflows by @carlopi in #6311
• CI Fix + match tests on less specific error messages by @Mytherin in #6320
• Fix #6314: select correct block index in IEJoin - and fix issues with left/right IE join resuming in case of multiple matches by @Mytherin in #6323
• CI: all workflows moved to nightly by @carlopi in #6334
• Fixes #6315: keep names/types around so description can be used after result is closed by @Mytherin in #6326
• Fix #5800: add missing Copy() calls, and add ALTERNATE_VERIFY method to verify Copy of INSERT/UPDATE/DELETE/COPY statements by @Mytherin in #6327
• Apply lower casing to extension aliases by @Mytherin in #6331
• Fix #6304: correctly handle NULL partitions and constant vectors, plus handle default parameters in COPY by @Mytherin in #6336
• [Python] DuckDBPyRelation: Change explain method and add sql method by @Tishj in #6287
• Fix Polars CI and properly implement check_ methods in the dataframes by @pdet in #6347
• Fixing a clang16 problem that slipped through by @hannes in #6345
• Fix #6341: LEFT/RIGHT/OUTER join on condition that is always true is only equal to a cross product if the other side is not empty by @Mytherin in #6342
• CI: Skip any CI on branches named 'feature' or 'master' by @carlopi in #6350
• Add correct bail-out to CSV auto-detection on oddly/inconsistently formatted CSV files by @Mytherin in #6330
• CI: Invert path-ignore for tools folders by @carlopi in #6353
• NULLs sort last in relational by @krlmlr in #5994
• Properly deal with Star (*) expressions in COPY ... (FORMAT JSON) by @lnkuiper in #6319
• fixes #6227 by @rpbouman in #6230
• fix typos in dictionary_store_worst_case.benchmark by @hnjylwb in #6371
• Julia: Support change timezone config by @xcaptain in #6358
• Paths-ignore on push by @carlopi in #6363
• JDBC - Add separate treatment for timestamptz values by @Jens-H in #6364
• bugfix: switch to fsspec's strip protocol impl by @Mause in #6361
• Disable tidy on ODBC for now by @Mytherin in #6379
• Implements function "sqlite3_column_table_name" for the sqlite3 wrapper by @TinyTinni in #6385
• [Python] No jemalloc for successful build on android by @papparapa in #6383
• throw BinderException on empty list in percentile by @samansmink in #6378
• Add optimizer flag to R and Python Substrait api by @LindsayWray in #6097
• fixes #6269 by @rpbouman in #6291
• Java: Use automatic resource management for AutoCloseable types by @MariusVolkhart in #6377
• Fix progress bar in (parallel) CSV reader by @Mytherin in #6397
• Fix #6393: for DESCRIBE order by column_index instead of column_name by @Mytherin in #6398
• ART (bug) fixes by @taniabogatsch in #6396
• [NodeJS] Support multi-statement prepare by @Tishj in #6278
• Java: Use StringBuilder where appropriate by @MariusVolkhart in #6373
• Bitpacking bug by @samansmink in #6402
• bugfix(fsspec): missing fs methods by @Mause in #6395
• Auto-load HTTPFS extension when http(s)/s3 files are queried and it is not loaded + upgrade SQLite scanner version/other extension fixes by @Mytherin in #6401
• Add helpful error message if a setting from an extension is attempted to be set when the extension is not loaded by @Mytherin in #6406
• namespace typos in blocking concurrent queue by @csruiliu in #6408
• Java: Implement DatabaseMetaData#isReadOnly() by @MariusVolkhart in #6375
• CI fixes by @carlopi in #6414
• Parquet: for DELTA_BYTE_ARRAY encoding verify that lengths of subsequent arrays do not exceed length of BYTE_ARRAY by @Mytherin in #6412
• Fix #6235: correctly return catalog for views in information_schema by @Mytherin in #6413
• Enable CMAKE_EXPORT_COMPILE_COMMANDS ON default by @JackDrogon in #6394
• Fix #5878: only delete the temp directory if we created it, otherwise delete only our temp files by @Mytherin in #6425
• Fix under-specified test by @Mytherin in #6419
• fix: logic fix to allow storage extension to implement DROP DATABASE by @stephaniewang526 in #6430
• Map bug combo of const & non-const lists by @LindsayWray in #6354
• Issue #6272: Window Scaled Repartitioning by @hawkfish in #6366
• respect column order for partitioned write by @samansmink in #6436
• Properly initialize string vector when reading large JSON arrays of strings by @lnkuiper in #6437
• Fix #6420 - correctly delete temporary files that are not explicitly read back but just dropped by @Mytherin in #6424
• Julia: support Pkg.test() by @chris-b1 in #6431
• fixes sqlite3_column_bytes nullptr access on some call ordering by @TinyTinni in #6409
• Write struct fields as optionally quoted in EXPORT DATABASE by @Tishj in #6416
• Enables sqlite3 wrapper tests for win32 builds by @TinyTinni in #6427
• Adding separate extension_directory configuration setting by @hannes in https://github.com/duckdb/duckdb/...

0.7.0 Preview Release "Labradorius"

This preview release of DuckDB is named "Labradorius" after the Labrador duck (Camptorhynchus labradorius) which was native to North America and went extinct in 1878 despite its reportedly bad taste.

Again, @Mytherin has written a blog post explaining the exciting list of new features in this release.

Binary builds are listed at the bottom of this post. Please note that it can take a couple of hours until binary builds for all platforms and environments are available.

Note: Again, this release introduces a backwards-incompatible change to the on-disk storage format. We suggest you use the EXPORT DATABASE command with the old version followed by IMPORT DATABASE with the new version to migrate your data. See the documentation for details.

What's Changed

• Use structs to avoid confusing C pointer wrappers by @krlmlr in #4961
• Enum type added to the types metadata table by @LindsayWray in #5290
• R: code format by @krlmlr in #5185
• Add starts_with function and operator by @papparapa in #5334
• Feature: Allow binary-formatted strings to be cast to integers by @Maxxen in #5337
• For range joins use NL join when the LHS or RHS side is tiny by @Mytherin in #5399
• Add support for LATERAL joins by @Mytherin in #5393
• [Julia] Add support for consuming a UNION vector into a DataFrame by @Tishj in #5360
• Issue #5314: At Time Zone by @hawkfish in #5341
• Decimal values now round when the value given has more decimals than the scale of the target by @Tishj in #5362
• Shell: add individual SQL queries to the history, instead of individual lines by @Mytherin in #5414
• Shell: add support for history search by @Mytherin in #5415
• Parallelise scanning result of ORDER_BY by @lnkuiper in #5403
• Add translate function by @zhouliqi in #5212
• Enable cmake to recognize AppleClang by @changhiskhan in #5432
• Support enum_code() function by @lokax in #5408
• Fix binder error and produce more informative error message. by @Tmonster in #5302
• Parquet Reader: Re-use (de)compression and dictionary buffers and allocate powers of two by @Mytherin in #5445
• Support RLE, DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY Parquet encodings by @Mytherin in #5457
• print profiling output for deserialized logical query plans by @ila in #5448
• Issue #5277: Sorted Aggregate Sorting by @hawkfish in #5456
• Add internal flag to duckdb_functions, and correctly set internal flag for internal functions by @Mytherin in #5462
• Add experimental R String passthrough support by @hannes in #5479
• Issue #5258: Quantile Negative Fractions by @hawkfish in #5463
• Arrow stream ingestion for JDBC client by @hannes in #5449
• PER_THREAD_OUTPUT flag for COPY by @hannes in #5412
• Feature: skip broken tests for now by @Mytherin in #5532
• Add Union All support to R extention by @Tmonster in #5484
• [Python] Add from_parquet features by @papparapa in #5492
• Add ExtractStatements to C API by @LindsayWray in #5524
• Improve http retry by @samansmink in #5549
• Issue #5277: Sorted Aggregate Window by @hawkfish in #5571
• Issue #5422: QUANTILE_DESC Decimals by @hawkfish in #5572
• Issue #5559: 2022g Time Zones by @hawkfish in #5570
• [Dev] Clean up of the python pkg folder structure by @Tishj in #5436
• httpfs: check environment vars for AWS Credentials by @satotake in #5419
• Misc union-type improvements by @Maxxen in #5617
• Fix so Left inner join doesn't re-optimize nodes by @Tmonster in #5620
• [Substrait] C API + from_substrait_json + bump on substrait version. by @pdet in #5613
• Allow strings in ColumnDataCollection to be written to disk by @lnkuiper in #5543
• [PythonDEV] Let clean.sh be run from anywhere, not just tools/pythonpkg by @Tishj in #5625
• Reorganize Join order optimizer code by @Tmonster in #5621
• [Catalog] Grab missing write_locks in a couple places by @Tishj in #5601
• Parquet info to Substrait by @pdet in #5627
• HTTP parquet optimizations by @samansmink in #5405
• Adding delta compression to Bitpacking compression by @samansmink in #5491
• [Python] Changed use of DuckDBPyConnection to shared_ptr by @Tishj in #5635
• Merge feature branch into master by @Mytherin in #5645
• [Python] Display progress bar by default in an interactive environment by @Tishj in #5596
• Add support for RESET statement on configuration options by @Tishj in #5603
• httpfs: Encode url path on request by @satotake in #5587
• Fix broken CI because of RESET statement by @Tishj in #5671
• Don't automatically set the bug label on issues by @Mytherin in #5680
• Add support for CREATE VIEW IF NOT EXISTS by @Mytherin in #5682
• Issue #5622: Validate Timezone Characters by @hawkfish in #5658
• Issue 5630 fix. by @Tmonster in #5644
• Adding COLUMN_TYPES option for read_csv_auto by @pdet in #5552
• [Python] Get rid of DuckDBPyResult (merged functionality into DuckDBPyRelation) by @Tishj in #5597
• feat: port nodejs tests to typescript by @Mause in #5632
• Improve nodejs README by @Tishj in #5688
• [Python] Add (partial) support for numpy.datetime64 objects by @Tishj in #5659
• retry on all httplib errors by @samansmink in #5684
• Return false if file doesn't exist by @Y-- in #5701
• Adding context option to not run replacement scans and exporting namespace of json substrait function - R by @pdet in #5689
• Issue #5609: Scope CTE Windows by @hawkfish in #5690
• Attempt to fix random NodeJS CI failure by @Tishj in #5710
• [Python] duckdb.execute() == duckdb.default_connection.execute() by @Tishj in #5650
• NodeJS: switch to using package_build, and add support to BUILD_NODE to Makefile by @Mytherin in #5691
• JDBC SNAPSHOT Jars by @hannes in #5687
• Fix NodeJS 19 CI for Windows by @Tishj in #5719
• Fix issue 5664 by @lokax in #5667
• Issue #5712: CURRENT_TIMESTAMP and CURRENT_TIME by @hawkfish in #5713
• [CSVReader] Catch a user error in supplying 'columns' option by @Tishj in #5721
• Improve suggestions when LOAD of an extension fails by @Mytherin in #5722
• doc(nodejs): amend arrow stream type docs by @Mause in #5731
• Fix for TSV throwing during sniffing by @pdet in #5555
• Statically link extensions on Linux with Clang by @jkub in #5653
• [Python] Add support for named parameters by @Tishj in #5611
• fix: nodejs source releases should be standalone by @Mause in #5734
• build: don't install python from chocolatey by @Mause in #5740
• fix: use non-string-splitting variable interpolation in binding.gyp.in by @Mause in #5745
• Equalizing DBConfig constructors by @nicku33 in #5747
• We should not treat replacement open paths as disk paths by @nicku33 in #5748
• Allow table in-out functions to be used in correlated subqueries and as LATERAL queries by @Mytherin in https://github.com/duckdb/duckdb...