ByConity 0.4.0 Release

Changelog

All notable changes to this project will be documented in this file.

[0.4.0] - 2024-04-15

🚀 Features

• Support MySQL Array Functions (FINAL)
• MySQL BI compatibility improvements
• proactively create table meta when creating table.
• support distributed outfile in worker
• Bucket size check && support alias column
• Add hdfs lock to concurrent outfile to the same file
• Close Lazy seek for hdfs
• Dedup worker only get wanted bucket numbers' parts
• KV Rbac performance optimization for granting lots of query to one user. Also fix a coredump and dead lock
• enable write part parallel
• make columns nullable by default under mysql
• fix array_remove and reverse function
• support state function no shuffle
• support additional-services for fulltext search.
• support delete bitmap cache in catalog
• json functions support Object(JSON)
• Support parts splitting for table scan
• support nullable date type for TTL
• upgrade aggregate function attributionAnalysis
• optimize query system tables/databases when tenant is set
• complete storage io trace and total query path
• enable microseconds event_time/query_start_time in worker query log
• Handle overflow date under MySQL dialect
• Disk partition writer size limit
• enhance skip index
• support distributed_aggregation_memory_efficient in optimizer
• Embed the query profiler into cnch binary
• statistics support implicit columns
• impl NVM load/store in PrimaryIndexCache
• support outfile to TOS
• add local disk fd cache

🐛 Bug Fixes

• remove validate during create , fix view rewrite , remove sync mv path
• Undo buffer use reversed transaction id to avoid write increment keys to bytekv
• fix isMonitoredCnchQuery core
• init lock request before put in map
• remove useless get delete bitmap request for non-unique table
• Fix for visiting the removed entry in remove operation
• fix insert select can not parallel on non-unique table
• when mv target table with materialized view throw exception
• get consul http host from multi env
• fix rbac check with wrong database name
• tenant should not see DBs create by default user
• fix global_disk_written_bytes leakage
• destroy task_record after phase-2 finished for merge task
• fix drop table meta and view tables core
• fix mv version part meta cache weight
• fix hdfsCloseFile() can be called twice if encounter error
• fix non partition based mv refresh
• need_invalid_cache also controls bitmaps cache.
• fix reading partition col without row count
• Fix table_definition_hash and make it backward compatible
• fix refresh materialized view submit async
• fix refresh empty database, vw settings, tenant id
• fix QueryAnalyzer process asterisk fail for INSERT query cnch dev
• fix assert failure in part gc thread
• fix file not found and add dlb inspector
• add supportsPrewhere check in pushFilterIntoStorage
• fix for materialized view union rewrite
• fix some functions can not push into prewhere
• enable single character of M-D h:m:s under mysql dialect
• fix unsafe request consul server
• enable runtime filter in bsp mode
• keep grace hash join when exchange type is broadcast
• fix write rows/bytes in query log
• catch finish senders in DiskExchangeManager::submitReadTask
• force UnifyNullableVisitor visit all plan nodes
• Fix arrayElement for map column under MySQL
• fix materialized view rewrite for unqiue table & view
• enable single character of M-D h:m:s under mysql dialect
• fix unsafe request consul server
• enable runtime filter in bsp mode
• add enable_add_projection_to_pruning setting
• keep grace hash join when exchange type is broadcast
• catch finish senders in DiskExchangeManager::submitReadTask
• force UnifyNullableVisitor visit all plan nodes
• Fix TableFinishStep overwrite partition when there is view rewrite
• Leader callback exception will be caught in storage elector
• add partition info to KV when seeing new staged part.
• Restrict tenanted users to whitelisted settings
• fix validateAndSyncBaseTablePartitions for query rewrite
• Fix hive table access and add enable_async_mv_debug setting
• unique table make sure parts and delete bitmap metas are same version
• Send dict table resources in beginning
• fix materialized view rewrite for order by columns
• enable implicit type convert and case insensitive by default under mysql dialect
• fix bug when grace hash join rehash
• fix core create table as using unique key
• update log level and ignore unlock exception
• fix bitmap index function pattern
• "X.Y.Z" format column query support in multi-tenant enviornment
• fix temporary_data_merge_threads set when enable optimizer
• make union input/output stream nullable data type consistent
• fix bound ratio incorrect merge cnch dev
• fix adddate for implicit type convert
• SingleDistinctAggregationToGroupBy support multiple input arguments
• support mark disk cache load data when disk cache write data
• support metrics2 for profile event
• disk cache support global setting
• getDeleteBitmap method need to be thread-safe
• optimizer handle non deterministic function better
• fix wrong write buffer usage in part dumper
• fix parse datetime best effort more than 20 fractional digits
• fix disk cache only cached partial data
• fix external agg spills too many small files
• reduce useless scatter when grace hash join
• Fix timediff function under mysql mode
• add lru cache for invert index to avoid oom AND disable String to Array(XXX) CAST at ddl modify column Scenario
• fix explain pipeline insert into select
• add table-level mutate settings
• touch active ts for all type tx type
• dialect type not set join use nulls cnch dev
• minor fix for quitting core bg threads
• fix first filter is const case in combineFilterEqualSize
• compatible settings format_csv_write_utf8_with_bom
• check disk partition writer finish result
• Split scheduler classes into files
• add untrack memroy for global memory tracker
• Add undo resource for fs lock
• remove path in fd_cache when DiskCacheLRU evict segments
• Use MultiWrite for writeUndoBuffer
• Cnch table's alter only acquire lock for DDL
• Allow BIT_AND and similar bitwise aggregation functions to operate on signed integers
• Only EnableLogging when send_logs_level is larger than information
• Fix MaxIOProfileEvents can't be logged if read from S3 without cache
• avoid return value from illegal address
• Fix S3 DeleteObjects exceed 1000 with error message MalformedXML
• fix report exception,See merge request dp/ClickHouse!19029
• fix lc fallback cut and compact error
• wait until finish for IO-bound tasks
• improve parts_info's accuracy for unique table.
• fix unclean drop in S3 storage policy
• fix RemoveRedundantSort with filling step
• narrow down the locality of scheduling in bsp mode, decouple scheduling trigger for bsp & mpp
• throw exception when get sample key if table not sample key
• fix cancel disk partition writer coredump
• Enable interactive transaction support when use optimizer

🧪 Testing

• Temporary disable 00715_bounding_ratio_ck233.sql
• Improve 50010_direct_insert test
• Testing remote function to community clickhouse
• Reenable 00715_bounding_ratio_ck233.sql
• Fix mysql integration test docker compose
• Add health check for mysql db in integration test
• Increase start period for healthcheck for mysql in integration test
• Debug compose integration
• Make mysql db init idempotent
• Remove debugging sleep
• Nightly integration test
• Fix nightly integration test
• Github workflow for nightly integration test
• Nightly test docker compose
• Nightly test github work flow
• Mysql integration test docker compose

⚙️ Miscellaneous Tasks

• Fix bin dir inside the docker
• Add script to create pr automatically
• Fix create_pr script
• Add split_pr script
• Add README and fix ci
• Fix util script
• Fix 50010_direct_insert
• Fix broken link in README.md
• Fix utils script
• Add action for auto release.
• Release disk space after pushing docker.