0.4.1-testonly-2

What's Changed

• [wip] prepare patches for 0.4.x release by @fanqi1909 in #1658
• cherry pick fixes for release/0.4.x by @fanqi1909 in #1659
• cherry-pick patches for 0.4.x by @fanqi1909 in #1660
• 0.4.x cp 05 13 by @fanqi1909 in #1669

Full Changelog: 0.4.0...0.4.1-testonly-2

ByConity 0.4.0 Release

Changelog

All notable changes to this project will be documented in this file.

[0.4.0] - 2024-04-15

🚀 Features

• Support MySQL Array Functions (FINAL)
• MySQL BI compatibility improvements
• proactively create table meta when creating table.
• support distributed outfile in worker
• Bucket size check && support alias column
• Add hdfs lock to concurrent outfile to the same file
• Close Lazy seek for hdfs
• Dedup worker only get wanted bucket numbers' parts
• KV Rbac performance optimization for granting lots of query to one user. Also fix a coredump and dead lock
• enable write part parallel
• make columns nullable by default under mysql
• fix array_remove and reverse function
• support state function no shuffle
• support additional-services for fulltext search.
• support delete bitmap cache in catalog
• json functions support Object(JSON)
• Support parts splitting for table scan
• support nullable date type for TTL
• upgrade aggregate function attributionAnalysis
• optimize query system tables/databases when tenant is set
• complete storage io trace and total query path
• enable microseconds event_time/query_start_time in worker query log
• Handle overflow date under MySQL dialect
• Disk partition writer size limit
• enhance skip index
• support distributed_aggregation_memory_efficient in optimizer
• Embed the query profiler into cnch binary
• statistics support implicit columns
• impl NVM load/store in PrimaryIndexCache
• support outfile to TOS
• add local disk fd cache

🐛 Bug Fixes

• remove validate during create , fix view rewrite , remove sync mv path
• Undo buffer use reversed transaction id to avoid write increment keys to bytekv
• fix isMonitoredCnchQuery core
• init lock request before put in map
• remove useless get delete bitmap request for non-unique table
• Fix for visiting the removed entry in remove operation
• fix insert select can not parallel on non-unique table
• when mv target table with materialized view throw exception
• get consul http host from multi env
• fix rbac check with wrong database name
• tenant should not see DBs create by default user
• fix global_disk_written_bytes leakage
• destroy task_record after phase-2 finished for merge task
• fix drop table meta and view tables core
• fix mv version part meta cache weight
• fix hdfsCloseFile() can be called twice if encounter error
• fix non partition based mv refresh
• need_invalid_cache also controls bitmaps cache.
• fix reading partition col without row count
• Fix table_definition_hash and make it backward compatible
• fix refresh materialized view submit async
• fix refresh empty database, vw settings, tenant id
• fix QueryAnalyzer process asterisk fail for INSERT query cnch dev
• fix assert failure in part gc thread
• fix file not found and add dlb inspector
• add supportsPrewhere check in pushFilterIntoStorage
• fix for materialized view union rewrite
• fix some functions can not push into prewhere
• enable single character of M-D h:m:s under mysql dialect
• fix unsafe request consul server
• enable runtime filter in bsp mode
• keep grace hash join when exchange type is broadcast
• fix write rows/bytes in query log
• catch finish senders in DiskExchangeManager::submitReadTask
• force UnifyNullableVisitor visit all plan nodes
• Fix arrayElement for map column under MySQL
• fix materialized view rewrite for unqiue table & view
• enable single character of M-D h:m:s under mysql dialect
• fix unsafe request consul server
• enable runtime filter in bsp mode
• add enable_add_projection_to_pruning setting
• keep grace hash join when exchange type is broadcast
• catch finish senders in DiskExchangeManager::submitReadTask
• force UnifyNullableVisitor visit all plan nodes
• Fix TableFinishStep overwrite partition when there is view rewrite
• Leader callback exception will be caught in storage elector
• add partition info to KV when seeing new staged part.
• Restrict tenanted users to whitelisted settings
• fix validateAndSyncBaseTablePartitions for query rewrite
• Fix hive table access and add enable_async_mv_debug setting
• unique table make sure parts and delete bitmap metas are same version
• Send dict table resources in beginning
• fix materialized view rewrite for order by columns
• enable implicit type convert and case insensitive by default under mysql dialect
• fix bug when grace hash join rehash
• fix core create table as using unique key
• update log level and ignore unlock exception
• fix bitmap index function pattern
• "X.Y.Z" format column query support in multi-tenant enviornment
• fix temporary_data_merge_threads set when enable optimizer
• make union input/output stream nullable data type consistent
• fix bound ratio incorrect merge cnch dev
• fix adddate for implicit type convert
• SingleDistinctAggregationToGroupBy support multiple input arguments
• support mark disk cache load data when disk cache write data
• support metrics2 for profile event
• disk cache support global setting
• getDeleteBitmap method need to be thread-safe
• optimizer handle non deterministic function better
• fix wrong write buffer usage in part dumper
• fix parse datetime best effort more than 20 fractional digits
• fix disk cache only cached partial data
• fix external agg spills too many small files
• reduce useless scatter when grace hash join
• Fix timediff function under mysql mode
• add lru cache for invert index to avoid oom AND disable String to Array(XXX) CAST at ddl modify column Scenario
• fix explain pipeline insert into select
• add table-level mutate settings
• touch active ts for all type tx type
• dialect type not set join use nulls cnch dev
• minor fix for quitting core bg threads
• fix first filter is const case in combineFilterEqualSize
• compatible settings format_csv_write_utf8_with_bom
• check disk partition writer finish result
• Split scheduler classes into files
• add untrack memroy for global memory tracker
• Add undo resource for fs lock
• remove path in fd_cache when DiskCacheLRU evict segments
• Use MultiWrite for writeUndoBuffer
• Cnch table's alter only acquire lock for DDL
• Allow BIT_AND and similar bitwise aggregation functions to operate on signed integers
• Only EnableLogging when send_logs_level is larger than information
• Fix MaxIOProfileEvents can't be logged if read from S3 without cache
• avoid return value from illegal address
• Fix S3 DeleteObjects exceed 1000 with error message MalformedXML
• fix report exception,See merge request dp/ClickHouse!19029
• fix lc fallback cut and compact error
• wait until finish for IO-bound tasks
• improve parts_info's accuracy for unique table.
• fix unclean drop in S3 storage policy
• fix RemoveRedundantSort with filling step
• narrow down the locality of scheduling in bsp mode, decouple scheduling trigger for bsp & mpp
• throw exception when get sample key if table not sample key
• fix cancel disk partition writer coredump
• Enable interactive transaction support when use optimizer

🧪 Testing

• Temporary disable 00715_bounding_ratio_ck233.sql
• Improve 50010_direct_insert test
• Testing remote function to community clickhouse
• Reenable 00715_bounding_ratio_ck233.sql
• Fix mysql integration test docker compose
• Add health check for mysql db in integration test
• Increase start period for healthcheck for mysql in integration test
• Debug compose integration
• Make mysql db init idempotent
• Remove debugging sleep
• Nightly integration test
• Fix nightly integration test
• Github workflow for nightly integration test
• Nightly test docker compose
• Nightly test github work flow
• Mysql integration test docker compose

⚙️ Miscellaneous Tasks

• Fix bin dir inside the docker
• Add script to create pr automatically
• Fix create_pr script
• Add split_pr script
• Add README and fix ci
• Fix util script
• Fix 50010_direct_insert
• Fix broken link in README.md
• Fix utils script
• Add action for auto release.
• Release disk space after pushing docker.

Testing release process

Testing

0.3.3-testonly.2

ci: add action for auto release.

0.3.2

Fix some issue:

• fix: make tso fallback configurable.
• fix: fix daemon manager unable to fetch status from servers
• fix: make select forwarding configurable [cp]
• fix: reportError variable lifetime
• fix: [cp] clear commit time for published part after dedup for unique table
• fix: set a valid enum status in parts_info_local.
• fix: Make show create cnch system tables works
• fix: unlock before call doRoundRobinPick
• fix: enable parallel Las files fetching
• fix: [cp] check_cancel for merge task & [cp] mark a 0 rows merged part as deleted when merging parts
• fix: [cp] limit bytes by the whole part chain
• fix: fix getMapKeys called multi times
• fix: remove unused code
• fix: fix UnifyNullableType don't handle cte
• fix: skip unstable test case for now
• fix: cover random join order
• fix: set enable_optimizer default true, set enable_auto_query_forwarding default true
• fix: fix wrong mapping in storage cache
• fix: fix txn coordinator NPE during server shutdown
• fix: fix txn coordinator NPE during server shutdown
• fix: fix wrong mapping in storage cache
• fix: fix join order by sort ordered_base_nodes stably.
• fix: Fix grouping set no merging
• fix: [cp cnch-ce-merge] minor fix preload for local table without uuid on worker side
• fix: fix not grouping key error
• fix: fix getMapKeys called multi times
• fix: ban optimizer when sql contains getMapKeys function
• fix: fix inner query execute when analyzing
• fix: try fix tealimit order by
• fix: fix the wrong bool expression.
• fix: fix undo buffer key conflict when dump parts parallel in multi workers [cp]
• fix: Fix attach parts with delta in s3
• fix: fix unstable testcase 10120
• fix: cherry pick from cnch1.4
• fix: cherry pick rewrite unknown left join identifier
• fix: remove unnecessary code [cp]
• fix: report real exception before execute
• fix: fix cnchhive alter cnch_server_vw
• fix: disable ut of bucket.eviction_expired
• fix: try fix random join order
• fix: Write empty object when no data
• fix: close outdated session
• fix: pick funcs/fixs
• fix: fix bug for low_cardinality_allow_in_native_format
• fix: fix gc under multi-server env
• fix: fix null pointer access for controller
• fix: Fix StartOfInterval Monotonic
• fix: cp memory leak fix
• fix: fix a bug that disk cache segment file may be incomplete
• fix: Fix ParseDatetime Coredump
• fix: Use MultiWrite for writeUndoBuffer & add flinkFieldHash fnv1aHash
• fix: Las integ
• fix: Fix Window Limit Push Down CNCH
• fix: [cp cnch-2.0] restart consume task if storage schema change
• fix: Fix Server UUID CNCH
• fix: try fix random join order
• fix: Fix transaction related issue on multi server environment
• fix: Subquery results are accessed by name
• fix: fix empty header core dump when create stats
• fix: remove SUBMIT_THREADPOOL for removeWorkerResource
• fix: fix coverity scan
• fix: column type in the header is inconsistent after ColumnPruning
• fix: pick the lock related code to avoid hold recursive mutex and request brpc mutex
• fix: [cp] avoid dead threads from ThreadPool
• fix: Fix Nullable Array Join for CNCH-2.0
• fix: refactor BitMap64 parsing
• fix: Prevent crash when passing wrong aggregation states to groupBitmap
• fix: parser ECNRYPT and SECURITY flags
• fix: add hive cnch_vw_read settings for back compatibility
• fix: optimizer uses cpu time to determine timeout
• fix: cherry-pick tso fix from 1.4
• fix: Fix function podName/getEnv
• fix: prefetch task core dump
• fix: Set default value for std::atomic to solve…
• fix: Add transaction clean tasks system tables and minor fix
• fix: fix failed to gc tables after drop database
• fix: fix optimizer join 1 equals 1
• fix: [cp] Fix aarch64 build for libunwind-old
• fix: update connection and retry when unable to access tso
• fix: rework query cache for optimizer
• fix: plan cache add current database check
• fix: [cp] Use old libunwind for jeprof backtrace
• fix: [cp] cnch-ce-merge temporary allow nullable type in map value
• fix: fix compatibility issue in TTL expression
• fix: wait for all threads finish before return to avoid use after free.
• fix: Update CMakeLists.txt to speed up compilation
• fix: union single property
• fix: fix a bug that tombstone may be trashed before covered parts
• fix: fix deadlock when shutdown context
• fix: [cp] cnch-ce-merge fix map parse null value
• fix: rewrite arrayElement to mapElement in optimizer

Add some feature:

• feat: merge-tree multi-stages reader
• feat: add information schema
• feat: remove duplicate or predicate
• feat: delay DiskExchangeSource read files op to generate
• feat: delay DiskExchangeSource read files op to generate
• feat:support json type
• feat: ReadBufferFromS3 Support SEEK_CUR mode
• feat: support aggregated funciton funnelPathSplit
• feat: refine topn filtering
• feat: improve executing PlanSemgent performace
• feat: Add metrics for updateTSO thread stop functioning
• feat: add plan segment query log
• feat: add send plan segment timeout
• feat: compatible Byte map (KV) with community map
• feat: Cnch ck23p3 alignment
• feat: [cp cnch-2.0] support to skip kafka offsets hole
• feat: complete AVRO format and support for CnchKafka in cnch-2.0
• feat: move insert_null_as_default from logical plan to physical plan
• feat: enable detach/drop staged part/partition command
• feat:[TO cnch-2.0] Support OUTFILE INTO OSS
• feat: [cp] set max merge concurrency based on number of workers & allow aggressive/final merge for small partition
• feat: Move simple reorder join into cascades
• feat: add optimizer profile events
• feat: fully support for insert_null_as_default
• feat: send bsp upstream exception to coordinator
• feat: Add configurable endpoint and timeout for vetos config

And add some optimization

0.3.1

Add some features and fix some bugs happen in 0.3.0:

fix: make global thread pool size same as before
fix: add PrimaryIndexCache back
fix:[breaking change]make the input & output of query exchange log right
feat: implement left (anti, semi, outer) , right(anti, semi, outer), inner join with unequal conditions
fix: use right fields of disk partition writer in query exchange log
fix: detach bsp cleanup from brpc worker
fix: fix index col cache miss
feat: control all bg threads
fix: temporary fix connection lifetime
fix: fix undo buffer clean in dm
feat: Use Read Buffer to replace Read Ahead when Infile from S3
feat: Use prefetch to optimize read from S3 external table performance
fix: skip adaptive load when index_granularities is empty & disable merge prefetcher for adaptive_granularity
feat: Add support for infile compression
fix: use replicated-join as default join shuffle, when statistics not exists
fix: fix potential coredump in getChecksums
fix: fix perf regression of sendResources
fix: fix unexpected while-loop when request empty s3 url
fix: fix create view bugs
fix: check close in BroadcastSenderProxy
fix: ensure worker clients in worker group are always ok
fix: disable rename column with bitmap index
feat: Use VW pool as the only way for getting clients (#126)
fix: cnch-2.0 fix map parse error pop back
fix: disable fsync for IndexFileWriter
fix: fix ALTER UPDATE
feat: Support multiple instances of same plan segment on same node in bsp mode
feat: refactor table scan step dyanmic cast
feat: Normalize the representation of nullable types in ANSI mode
fix: Remove alias checking in ANSI mode
fix: explain add verbose setting
fix: fix array join with lambda
fix: fix in database.table is not supported by optimizer cnch-ce-merge
fix: fix settings in global context changed after startup
fix: [cp cnch-ce-merge] add some setting params to compatible with 1.4
fix: Fix write to gin without any term
fix: some fix & opitmize for bitmap index
fix: after cnch-2.0.3 release branch
fix: fix forwarding logic
fix: fix performance regression
fix: disk partition writer enable sender metrics by default
fix: enable bsp_mode in explain analyze
fix: sync recent fixs to cnch-2.0 from cnch-1.4
fix: ban rewrite_map_col in optimizer link
feat: show bytehouse logo when connect to client
fix: refactor commit parts and txn commit logic
fix: fix refreshWhere getTable twice
feat: Change the default value of use_environment_credentials
fix: plan cache add stats check
feat: Use IRSA / credential files for S3 Disk
fix: nerver set false to partition_metrics_loaded.
feat: generate actively prewhere by statistics
fix: fix cmake copy bug
fix: fix core for dedup worker
feat: Disk partition writer exchange log
feat: Cherry pick any_value CNCH
feat: disable enable_auto_query_forwarding
fix: fix aws log level can't be set
fix: add optimizer unsupported setting list
fix: async send for bsp register
fix: fix ctes in view
fix: cnch 2.0 fix read virtual column _map_column_keys
fix: fix distinct to agg
fix: fix client display progress protocol compatibility with 2.0 itself
feat: add disk_cache_read_bytes stats to query log for totally query path

0.3.0

Add some features and optimisations. Fixing bugs and refactor code.

New features:

• Using prefetch and async read to optimize remote disk read
• Attempt to add memory tracking to CRoaring
• explain metadata add fucntion_const_arguments field
• optimize PlanSegmentProcessList mutex
• remove add exchange
• integrate nvm_cache into checksums_cache
• refactor table scan step dyanmic cast
• background gc for bsp
• Fix RBAC Entity to SQL Conversion Logic
• swap join order plan hint
• Support setting to convert MergeTree to CnchMergeTree
• add dry_run for InterpretSelectQuery
• update croaring version to 0.9.9
• add brpc_data_parts_timeout_ms parameters to control part data sending timeout
• add brpc_data_parts_timeout_ms parameters to control part data sending timeout
• fulltext search cnch ce merge pick
• optimizer support create select
• Support Division By Zero
• Analyzer constant folding
• (with the fix) Extend the support of interpreting "WindowFunction + OrderBy"
• Support DB/Table Snapshots
• RBAC
• use threadpool for worker service
• support bitmap index in cnch2.0
• 【CP】add some compatible settings for 2.0(alpha、beta)
• nvm_cache persist & recover
• Optimization for log and yield in leader election component
• add hybrid allocation table settings for backward compatibility
• To cnch-2.0] Support configurable virtual_hosted_style for s3 infile
• use real pk fk info in join estimator
• nvm_cache impl and integrate with mark_cache & uncompressed_cache
• [cherry]S3 Storage support
• [cp] support reload log level from config file
• Update aws-cpp-sdk and submodules, Using CrtHttpClient
• update statistics binary for plan check cnch-ce-merge
• common expression share
• support protobuf for QueryPlan, skip input_streams for JoinStep hash_equals
• explain analyze pipeline
• fix errors in debug mode
• staged execution and disk based shuffle
• clean up all warnings and treat warning as error
• workload dump to client
• CnchPartsInfo and cnchTrashItemsInfo with snapshots and recalculation.
• support explain json in full format
• Log segment profiles into cnch_query_log and query_log
• DELETE Mutation
• cherry pick push partition key condition down to storage
• cherry pick Add ability to pause/resume clustering task
• fix range filter inclusive bug
• rm unnecessary copy of plan segment
• CnchMaterializingMySQL 2.0 support
• 【CP】window function parallel evaluation cherry from community
• 【CP】replace exchange_timeout with max_execution_time
• API handler for resource reporting
• use new function register API
• Write graphviz dot strings into query_log when print_graph=1
• [CP]support split disk cache into meta&data for cnch-2.0
• Add partition to the system part_log and merge table.
• cp backup vw to cnch2.0
• Add support for DWARF-5
• introduce StorageDictCloudMergeTree
• Add Decimal 256 Support CNCH
• bitengine basic create table, write and read
• optimizer support offset
• Support agg window keys powerset in property
• unique table support attach/detach commands when using S3 storage
• create view check column names
• fix checkSubQueryReturnSingleRow nullptr core
• unique table support bucket lock & fix modify cluster by key bug
• unique table support bucket lock & fix modify cluster by key bug
• Export metric to prometheus
• cherry-pick workload tuning advisor to cnch

v0.2.2-fix.1

Fix get machine ip incorrectly in CentOS

0.2.2

New Features

1. Users now could access data managed by hive and glue without creating external tables. (#686)
2. The optimizer could collect statistics automatically.

Bug Fix

1. Fixed rename failure issue (#676).
2. Fixed slowness in merge and kafka write.
3. Fixed unfunctional bloom filter (#694).
4. Fixed transaction-releated issues in (#657 and #624)

0.2.0

New Feature

• Add multi-workers local cluster
• Add scripts for packaging
• StorageCnchMergeTree properly supports skipping indices
• Enable data preload write through
• support krb5 for hdfs
• cnchhive support read ORC format (preview)
• add certificate test suite
• add sql test suite
• enable multi-worker in ci
• auto forward query to host server when having multiple servers
• patch S3 storage basic
• add hms with kerberos
• add timeout for FoundationDB transactions
• use optimizer for join by default
• Fix unstable test cases
• support expressions with window functions
• add hive docker compose
• ByConity Projection
• add docker image build for centos
• improve ingest column query
• support query cache
• Add S3 rename tool
• support query result cache
• support key value store for RBAC

Bug Fix

• Fix read without cache
• Remove buffer size assertion during loading marks
• Fix attach related issue
• Sync from internal - fix optimizer arrayjoin with limit issue
• Sync from internal: Statistics related fix
• Fix txn leak when exception happening
• Core fix for querying system tables
• Fix thread map related issue
• Directly remove part from catalog if part don't exists on disk
• Fix fmt cannot write to file
• add calculate column sizes
• Fix mark count missmatch during loading index
• Fix partition prunning with ttl when ttl expression doesn't exactly match partition expression
• rpc service use ipv6
• Fix implicit map data loss after merges
• dns service discovery mode not work in k8s
• rpc service use ipv6
• update config file to support ipv6
• Explicitly disable optimizer for insert select
• fix test case 00921_datetime64_compatibility_long
• fix hang when shutdown
• CertificateStateful not reporting failure
• fix test case in sql suite
• fix some test cases
• fix some join test case
• fix bad a test case
• fix listen_try logic on brpc
• fix show create database query
• Support to read Hive table without partition key
• Complete the full support of positional arguments for Analyzer
• support listen_host and listen try for TSO, DM, RM
• Update Dockerfile
• fix forward query to worker
• Fix Analyzer over window functions + order by
• fix unstable test case
• fix Fdb scanning timeout issue
• fix failed to get exchange port
• fix executable wrapper script
• fix submodule
• fix submodule cassandra
• move setings to CI
• fix document
• fix insert data to VersionedCollapsingMergeTree get final error
• add rm for executable wrapper
• fix dictionary failed to update and tenant id
• fix query cache
• fix coredump related to query cache result
• fix query forwarding

Performance

• Performance optimization for key condition analysis
• Performance optimization + part removal fix
• Performance optimization for count distinct
• Performance optimization: trivial count
• Increased parallelism of group by
• Reduce syscalls and catalog calls
• Remove inefficient call to database catalog
• Precalculate values inside HashTableGrower
• In-memory cache for primary key index
• Predict hash table size for aggregation
• Optimize optimizer static cost

Enhancement

• bypass db lock when creating table by default
• Use VW pool as the only way for getting clients
• Update local deployment
• Update docker image for executable wrapper
• Improve CAS operation
• Remove use test from test cases
• Support hive table on external Hdfs
• support get worker id info from config
• old kernel hdfs support
• Added the first version of release_note generation script
• add annotations for FunctionsConversion
• hive schema inference
• Export s3's virtual host style config

Test

• Update clickhouse-test
• Testing test case client
• add ci cases

Documentation

• Update Readme.md
• Discord and community section added to ReadMe
• Clean up the deployment READMEs
• Remove outdated content
• Update README.md with new contribution guideline