-
Notifications
You must be signed in to change notification settings - Fork 6.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement Dynamic data type #63058
Implement Dynamic data type #63058
Conversation
…nto dynamic-data-type
…olumns in wide parts
This is an automated comment for commit 48cab9e with description of existing statuses. It's updated for the latest CI running ❌ Click here to open a full report in a separate page
Successful checks
|
… dynamic columns, fix style errors and tests
|
It works well for me (with release build downloaded from the CI): https://pastila.nl/?00029bb2/ef837fea9f5ea85db2f5edfb5a6f5ab1#YdWQhOS+QIaDkfwyhvftSA== In interactive mode works too: https://pastila.nl/?0001557a/19a1bd199f27efef1253e18504146390#wVSn7Qee2pJN02q0RTgV6w== Insert takes around 0.5 sec |
Yes, problem was in debug build Two things here:
|
Dynamic stores Variant inside, so we just propogated
Similar to |
Can't filter by int because of types mismatch.
|
It's expected, value of Dynamic column can be compared only with value of another Dynamic column, otherwise we don't know with wich type to compare. Please, see documentation (there is also a note how to filter): |
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
30dce78
…e4eea94f975a6cb9847d497f0cbd8 Cherry pick #63058 to 24.5: Implement Dynamic data type
Backport #63058 to 24.5: Implement Dynamic data type
* [GLUTEN-1632][CH]Daily Update Clickhouse Version (20240524) * Fix build due to ClickHouse/ClickHouse#63058 --------- Co-authored-by: kyligence-git <gluten@kyligence.io> Co-authored-by: Chang Chen <baibaichen@gmail.com>
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Implement Dynamic data type that allows to store values of any type inside it without knowing all of them in advance.
Dynamic type is available under a setting
allow_experimental_dynamic_type
.Reference: #54864
Documentation entry for user-facing changes
Dynamic
This type allows to store values of any type inside it without knowing all of them in advance.
To declare a column of
Dynamic
type, use the following syntax:Where
N
is an optional parameter between1
and255
indicating how many different data types can be stored inside a column with typeDynamic
across single block of data that is stored separately (for example across single data part forMergeTree
table). If this limit is exceeded, all new types will be converted to typeString
. Default value ofmax_types
is32
.Creating Dynamic
Using
Dynamic
type in table column definition:Using CAST from ordinary column:
Using CAST from
Variant
column:Reading Dynamic nested types as subcolumns
Dynamic
type supports reading a single nested type from aDynamic
column using the type name as a subcolumn.So, if you have column
d Dynamic
you can read a subcolumn of any valid typeT
using syntaxd.T
,this subcolumn will have type
Nullable(T)
ifT
can be insideNullable
andT
otherwise. This subcolumn willbe the same size as original
Dynamic
column and will containNULL
values (or empty values ifT
cannot be insideNullable
)in all rows in which original
Dynamic
column doesn't have typeT
.Dynamic
subcolumns can be also read using functiondynamicElement(dynamic_column, type_name)
.Examples:
To know what variant is stored in each row function
dynamicType(dynamic_column)
can be used. It returnsString
with value type name for each row (or'None'
if row isNULL
).Example:
Reading Dynamic type from the data
All text formats (TSV, CSV, CustomSeparated, Values, JSONEachRow, etc) supports reading
Dynamic
type. During data parsing ClickHouse tries to infer the type of each value and use it during insertion toDynamic
column.Example: