Changelog
All notable changes to this project will be documented in this file.
[0.4.0] - 2024-04-15
🚀 Features
- Support MySQL Array Functions (FINAL)
- MySQL BI compatibility improvements
- proactively create table meta when creating table.
- support distributed outfile in worker
- Bucket size check && support alias column
- Add hdfs lock to concurrent outfile to the same file
- Close Lazy seek for hdfs
- Dedup worker only get wanted bucket numbers' parts
- KV Rbac performance optimization for granting lots of query to one user. Also fix a coredump and dead lock
- enable write part parallel
- make columns nullable by default under mysql
- fix array_remove and reverse function
- support state function no shuffle
- support additional-services for fulltext search.
- support delete bitmap cache in catalog
- json functions support Object(JSON)
- Support parts splitting for table scan
- support nullable date type for TTL
- upgrade aggregate function attributionAnalysis
- optimize query system tables/databases when tenant is set
- complete storage io trace and total query path
- enable microseconds event_time/query_start_time in worker query log
- Handle overflow date under MySQL dialect
- Disk partition writer size limit
- enhance skip index
- support distributed_aggregation_memory_efficient in optimizer
- Embed the query profiler into cnch binary
- statistics support implicit columns
- impl NVM load/store in PrimaryIndexCache
- support outfile to TOS
- add local disk fd cache
🐛 Bug Fixes
- remove validate during create , fix view rewrite , remove sync mv path
- Undo buffer use reversed transaction id to avoid write increment keys to bytekv
- fix isMonitoredCnchQuery core
- init lock request before put in map
- remove useless get delete bitmap request for non-unique table
- Fix for visiting the removed entry in remove operation
- fix insert select can not parallel on non-unique table
- when mv target table with materialized view throw exception
- get consul http host from multi env
- fix rbac check with wrong database name
- tenant should not see DBs create by default user
- fix global_disk_written_bytes leakage
- destroy task_record after phase-2 finished for merge task
- fix drop table meta and view tables core
- fix mv version part meta cache weight
- fix hdfsCloseFile() can be called twice if encounter error
- fix non partition based mv refresh
- need_invalid_cache also controls bitmaps cache.
- fix reading partition col without row count
- Fix table_definition_hash and make it backward compatible
- fix refresh materialized view submit async
- fix refresh empty database, vw settings, tenant id
- fix QueryAnalyzer process asterisk fail for INSERT query cnch dev
- fix assert failure in part gc thread
- fix file not found and add dlb inspector
- add supportsPrewhere check in pushFilterIntoStorage
- fix for materialized view union rewrite
- fix some functions can not push into prewhere
- enable single character of M-D h:m:s under mysql dialect
- fix unsafe request consul server
- enable runtime filter in bsp mode
- keep grace hash join when exchange type is broadcast
- fix write rows/bytes in query log
- catch finish senders in DiskExchangeManager::submitReadTask
- force UnifyNullableVisitor visit all plan nodes
- Fix arrayElement for map column under MySQL
- fix materialized view rewrite for unqiue table & view
- enable single character of M-D h:m:s under mysql dialect
- fix unsafe request consul server
- enable runtime filter in bsp mode
- add enable_add_projection_to_pruning setting
- keep grace hash join when exchange type is broadcast
- catch finish senders in DiskExchangeManager::submitReadTask
- force UnifyNullableVisitor visit all plan nodes
- Fix TableFinishStep overwrite partition when there is view rewrite
- Leader callback exception will be caught in storage elector
- add partition info to KV when seeing new staged part.
- Restrict tenanted users to whitelisted settings
- fix validateAndSyncBaseTablePartitions for query rewrite
- Fix hive table access and add enable_async_mv_debug setting
- unique table make sure parts and delete bitmap metas are same version
- Send dict table resources in beginning
- fix materialized view rewrite for order by columns
- enable implicit type convert and case insensitive by default under mysql dialect
- fix bug when grace hash join rehash
- fix core create table as using unique key
- update log level and ignore unlock exception
- fix bitmap index function pattern
- "X.Y.Z" format column query support in multi-tenant enviornment
- fix temporary_data_merge_threads set when enable optimizer
- make union input/output stream nullable data type consistent
- fix bound ratio incorrect merge cnch dev
- fix adddate for implicit type convert
- SingleDistinctAggregationToGroupBy support multiple input arguments
- support mark disk cache load data when disk cache write data
- support metrics2 for profile event
- disk cache support global setting
- getDeleteBitmap method need to be thread-safe
- optimizer handle non deterministic function better
- fix wrong write buffer usage in part dumper
- fix parse datetime best effort more than 20 fractional digits
- fix disk cache only cached partial data
- fix external agg spills too many small files
- reduce useless scatter when grace hash join
- Fix timediff function under mysql mode
- add lru cache for invert index to avoid oom AND disable String to Array(XXX) CAST at ddl modify column Scenario
- fix explain pipeline insert into select
- add table-level mutate settings
- touch active ts for all type tx type
- dialect type not set join use nulls cnch dev
- minor fix for quitting core bg threads
- fix first filter is const case in combineFilterEqualSize
- compatible settings format_csv_write_utf8_with_bom
- check disk partition writer finish result
- Split scheduler classes into files
- add untrack memroy for global memory tracker
- Add undo resource for fs lock
- remove path in fd_cache when DiskCacheLRU evict segments
- Use MultiWrite for writeUndoBuffer
- Cnch table's alter only acquire lock for DDL
- Allow BIT_AND and similar bitwise aggregation functions to operate on signed integers
- Only EnableLogging when send_logs_level is larger than information
- Fix MaxIOProfileEvents can't be logged if read from S3 without cache
- avoid return value from illegal address
- Fix S3 DeleteObjects exceed 1000 with error message MalformedXML
- fix report exception,See merge request dp/ClickHouse!19029
- fix lc fallback cut and compact error
- wait until finish for IO-bound tasks
- improve parts_info's accuracy for unique table.
- fix unclean drop in S3 storage policy
- fix RemoveRedundantSort with filling step
- narrow down the locality of scheduling in bsp mode, decouple scheduling trigger for bsp & mpp
- throw exception when get sample key if table not sample key
- fix cancel disk partition writer coredump
- Enable interactive transaction support when use optimizer
🧪 Testing
- Temporary disable 00715_bounding_ratio_ck233.sql
- Improve 50010_direct_insert test
- Testing remote function to community clickhouse
- Reenable 00715_bounding_ratio_ck233.sql
- Fix mysql integration test docker compose
- Add health check for mysql db in integration test
- Increase start period for healthcheck for mysql in integration test
- Debug compose integration
- Make mysql db init idempotent
- Remove debugging sleep
- Nightly integration test
- Fix nightly integration test
- Github workflow for nightly integration test
- Nightly test docker compose
- Nightly test github work flow
- Mysql integration test docker compose
⚙️ Miscellaneous Tasks
- Fix bin dir inside the docker
- Add script to create pr automatically
- Fix create_pr script
- Add split_pr script
- Add README and fix ci
- Fix util script
- Fix 50010_direct_insert
- Fix broken link in README.md
- Fix utils script
- Add action for auto release.
- Release disk space after pushing docker.