Refactoring to perform cardinality estimation specifically for YT. #4482

alephonea · 2024-05-13T22:24:06Z

Reviewed in
#3866

but accidentally closed that PR - reopening.

github-actions · 2024-05-13T22:27:36Z

⚪ 2024-05-13 22:27:36 UTC Pre-commit check for 44833f2 has started.
⚪ 2024-05-13 22:27:38 UTC Build linux-x86_64-relwithdebinfo is running...
🔴 2024-05-13 22:44:20 UTC Build failed. see the build logs.
🔴 2024-05-13 22:45:58 UTC Tests run skipped.

github-actions · 2024-05-13T22:27:50Z

⚪ 2024-05-13 22:27:50 UTC Pre-commit check for 44833f2 has started.
⚪ 2024-05-13 22:27:53 UTC Build linux-x86_64-release-clang14 is running...
🔴 2024-05-13 22:43:42 UTC Build failed. see the build logs.

github-actions · 2024-05-13T22:27:58Z

⚪ 2024-05-13 22:27:58 UTC Pre-commit check for 44833f2 has started.
⚪ 2024-05-13 22:28:00 UTC Build linux-x86_64-release-asan is running...
🔴 2024-05-13 22:44:34 UTC Build failed. see the build logs.
🔴 2024-05-13 22:46:08 UTC Tests run skipped.

github-actions · 2024-05-14T16:06:12Z

⚪ 2024-05-14 16:06:11 UTC Pre-commit check for 0d36e1c has started.
⚪ 2024-05-14 16:06:14 UTC Build linux-x86_64-release-clang14 is running...
🟢 2024-05-14 16:35:59 UTC Build successful.

github-actions · 2024-05-14T16:06:13Z

⚪ 2024-05-14 16:06:12 UTC Pre-commit check for 0d36e1c has started.
⚪ 2024-05-14 16:06:15 UTC Build linux-x86_64-release-asan is running...
🟢 2024-05-14 16:22:58 UTC Build successful.
⚪ 2024-05-14 16:24:45 UTC Tests are running...
🔴 2024-05-14 18:12:14 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
14057	13966	0	32	50	9

github-actions · 2024-05-14T16:24:08Z

⚪ 2024-05-14 16:24:08 UTC Pre-commit check for 0d36e1c has started.
⚪ 2024-05-14 16:24:09 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-05-14 16:42:34 UTC Build successful.
⚪ 2024-05-14 16:44:30 UTC Tests are running...
🔴 2024-05-14 18:37:59 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
72892	59685	0	3	13188	16

rvu1024 · 2024-05-15T15:38:35Z

ydb/library/yql/core/yql_statistics.h

    TOptimizerStatistics() : KeyColumns(EmptyColumns) {}
    TOptimizerStatistics(double nrows, int ncols): Nrows(nrows), Ncols(ncols), KeyColumns(EmptyColumns) {}
    TOptimizerStatistics(double nrows, int ncols, double cost): Nrows(nrows), Ncols(ncols), Cost(cost), KeyColumns(EmptyColumns) {}
    TOptimizerStatistics(EStatisticsType type, double nrows, int ncols, double cost): Type(type), Nrows(nrows), Ncols(ncols), Cost(cost), KeyColumns(EmptyColumns) {}
    TOptimizerStatistics(EStatisticsType type, double nrows, int ncols, double byteSize, double cost): Type(type), Nrows(nrows), Ncols(ncols), ByteSize(byteSize), Cost(cost), KeyColumns(EmptyColumns) {}
    TOptimizerStatistics(EStatisticsType type, double nrows, int ncols, double cost, const TVector<TString>& keyColumns): Type(type), Nrows(nrows), Ncols(ncols), Cost(cost), KeyColumns(keyColumns) {}
    TOptimizerStatistics(EStatisticsType type, double nrows, int ncols, double byteSize, double cost, const TVector<TString>& keyColumns): Type(type), Nrows(nrows), Ncols(ncols), ByteSize(byteSize), Cost(cost), KeyColumns(keyColumns) {}
+    TOptimizerStatistics(EStatisticsType type, double nrows, int ncols, double byteSize, double cost, const TVector<TString>& keyColumns, IProviderStatistics* specific)


Почему нельзя все сделать одним конструктором с дефолтными значениями аргументов?

rvu1024 · 2024-05-15T15:40:41Z

ydb/library/yql/core/yql_cost_function.h

+    Undefined,
+    LookupJoin,
+    MapJoin,
+    GraceJoin,
+    StreamLookupJoin //Right part can be updated during an operation. Used mainly for joining streams with lookup tables. Currently impplemented in Dq by LookupInputTransform


Здесь перечислены только DQ-специфичные алгоритмы. В YT провайдере есть другие реализации. Что с ними предполагается делать? Как будет расширяться этот enum?

Предполагается что enum будет общим для DQ и YT. Если YT понадобится алгоритм джойна, который здесь не перечислен, то для DQ IsJoinApplicable() должен вернуть false для этого алгоритма.

Сейчас YT делает либо Map join, либо merge join (GraceJoin в этом енуме).

GraceJoin больше соответствует Common стратегии. Не понятно, как тут различать merge и common

Добавил MergeJoin

В качестве альтернативы, можно привязать алгоритмы все джойна к системе, в которой они выполняются:

DqLookupJoin,
DqMapJoin,
..
YtMapJoin,
YtMergeJoin

чтобы было понятно из значения enum-а как конкретно он будет исполняться. Но мне кажется, это излишнее усложнение.

rvu1024 · 2024-05-15T15:41:18Z

ydb/library/yql/core/yql_cost_function.h

+    TJoinColumn(TString relName, TString attributeName) : RelName(relName),
        AttributeName(attributeName) {}


std::move()

rvu1024 · 2024-05-15T15:43:28Z

ydb/library/yql/core/yql_cost_function.cpp

@@ -37,7 +23,7 @@ bool NDq::operator < (const NDq::TJoinColumn& c1, const NDq::TJoinColumn& c2) {
    return false;
 }

-TString NYql::ConvertToJoinAlgoString(EJoinAlgoType joinAlgo) {
+TString ConvertToJoinAlgoString(EJoinAlgoType joinAlgo) {


Используй GENERATE_ENUM_SERIALIZATION в ya.make

Эта функция используется для вывода в std::stringstream (как и ConvertToJoinString) который не поддерживается GENERATE_ENUM_SERIALIZATION

Не понял про поддержку. Ты будешь просто писать ToString(joinAlgo) вместо ConvertToJoinAlgoString(joinAlgo). И ToString() тебе сгенерит GENERATE_ENUM_SERIALIZATION, а не руками перечислять все константы

rvu1024 · 2024-05-15T15:45:05Z

ydb/library/yql/core/cbo/cbo_optimizer_new.h

+        const std::set<std::pair<NDq::TJoinColumn, NDq::TJoinColumn>>& joinConditions,
+        EJoinAlgoType joinAlgo) const override;
+
+    static const TBaseProviderContext& instance();


Style: Instance

rvu1024 · 2024-05-15T15:46:13Z

ydb/library/yql/core/cbo/cbo_optimizer_new.cpp

@@ -109,4 +109,123 @@ void TJoinOptimizerNode::Print(std::stringstream& stream, int ntabs) {
    RightArg->Print(stream, ntabs+1);
 }

+bool IsPKJoin(const TOptimizerStatistics& stats, const TVector<TString>& joinKeys) {
+    if (stats.KeyColumns.size()==0) {


Style: пробелы вокруг ==

rvu1024 · 2024-05-15T15:46:45Z

ydb/library/yql/core/cbo/cbo_optimizer_new.cpp

+        return false;
+    }
+
+    for(size_t i=0; i<stats.KeyColumns.size(); i++){


Style: пробелы вокруг =, < и ()

github-actions · 2024-05-15T22:21:51Z

⚪ 2024-05-15 22:21:50 UTC Pre-commit check for 667db86 has started.
⚪ 2024-05-15 22:21:53 UTC Build linux-x86_64-release-clang14 is running...
🟢 2024-05-15 22:37:04 UTC Build successful.

github-actions · 2024-05-15T22:21:57Z

⚪ 2024-05-15 22:21:57 UTC Pre-commit check for 667db86 has started.
⚪ 2024-05-15 22:21:59 UTC Build linux-x86_64-release-asan is running...
🟢 2024-05-15 22:39:03 UTC Build successful.
⚪ 2024-05-15 22:41:00 UTC Tests are running...
🔴 2024-05-16 00:32:07 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
14365	14263	0	42	51	9

github-actions · 2024-05-15T22:21:59Z

⚪ 2024-05-15 22:21:58 UTC Pre-commit check for 667db86 has started.
⚪ 2024-05-15 22:22:01 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-05-15 22:38:26 UTC Build successful.
⚪ 2024-05-15 22:40:27 UTC Tests are running...
🔴 2024-05-16 00:36:38 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
73204	60533	0	6	12657	8

github-actions · 2024-05-21T22:07:37Z

⚪ 2024-05-21 22:07:37 UTC Pre-commit check for da0f35c has started.
⚪ 2024-05-21 22:07:40 UTC Build linux-x86_64-release-clang14 is running...
🟢 2024-05-21 22:23:51 UTC Build successful.

github-actions · 2024-05-21T22:07:53Z

⚪ 2024-05-21 22:07:52 UTC Pre-commit check for da0f35c has started.
⚪ 2024-05-21 22:07:55 UTC Build linux-x86_64-release-asan is running...
🟢 2024-05-21 22:24:18 UTC Build successful.
⚪ 2024-05-21 22:26:08 UTC Tests are running...
🔴 2024-05-22 00:23:16 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
14439	14335	0	42	50	12

github-actions · 2024-05-21T22:10:05Z

⚪ 2024-05-21 22:10:05 UTC Pre-commit check for da0f35c has started.
⚪ 2024-05-21 22:10:07 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-05-21 22:27:00 UTC Build successful.
⚪ 2024-05-21 22:28:55 UTC Tests are running...
🔴 2024-05-22 00:24:13 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
73414	60714	0	3	12692	5

github-actions · 2024-05-22T18:43:27Z

⚪ 2024-05-22 18:43:26 UTC Pre-commit check for 841ded6 has started.
⚪ 2024-05-22 18:43:28 UTC Build linux-x86_64-release-asan is running...
⚫ 2024-05-22 18:53:50 UTC Check cancelled

github-actions · 2024-05-22T18:44:57Z

⚪ 2024-05-22 18:44:57 UTC Pre-commit check for 841ded6 has started.
⚪ 2024-05-22 18:44:59 UTC Build linux-x86_64-release-clang14 is running...
⚫ 2024-05-22 18:54:06 UTC Check cancelled

github-actions · 2024-05-22T18:45:36Z

⚪ 2024-05-22 18:45:35 UTC Pre-commit check for 841ded6 has started.
⚪ 2024-05-22 18:45:38 UTC Build linux-x86_64-relwithdebinfo is running...
⚫ 2024-05-22 18:54:07 UTC Check cancelled

github-actions · 2024-05-22T18:55:18Z

⚪ 2024-05-22 18:55:18 UTC Pre-commit check for e23d56f has started.
⚪ 2024-05-22 18:55:20 UTC Build linux-x86_64-release-clang14 is running...
🟢 2024-05-22 19:00:18 UTC Build successful.

github-actions · 2024-05-22T18:57:36Z

⚪ 2024-05-22 18:57:36 UTC Pre-commit check for e23d56f has started.
⚪ 2024-05-22 18:57:39 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-05-22 19:05:01 UTC Build successful.
⚪ 2024-05-22 19:06:52 UTC Tests are running...
🔴 2024-05-22 21:03:28 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
73449	60739	0	5	12695	10

github-actions · 2024-05-22T18:57:36Z

⚪ 2024-05-22 18:57:36 UTC Pre-commit check for e23d56f has started.
⚪ 2024-05-22 18:57:39 UTC Build linux-x86_64-release-asan is running...
🟢 2024-05-22 19:03:18 UTC Build successful.
⚪ 2024-05-22 19:05:08 UTC Tests are running...
🔴 2024-05-22 21:06:50 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
14450	14342	0	44	55	9

alephonea requested review from a team as code owners May 13, 2024 22:24

pavelvelikhov previously approved these changes May 14, 2024

View reviewed changes

alephonea dismissed pavelvelikhov’s stale review via e70ace7 May 14, 2024 16:02

rvu1024 requested changes May 15, 2024

View reviewed changes

alephonea force-pushed the cbo-exp1 branch from e70ace7 to 14be4cc Compare May 15, 2024 22:18

alephonea requested a review from a team as a code owner May 15, 2024 22:18

alephonea force-pushed the cbo-exp1 branch 2 times, most recently from e277117 to 14c68ba Compare May 21, 2024 22:05

alephonea force-pushed the cbo-exp1 branch from 14c68ba to 70bb449 Compare May 22, 2024 18:41

Refactoring to perform cardinality estimation specifically for YT.

bea3bec

alephonea force-pushed the cbo-exp1 branch from 70bb449 to bea3bec Compare May 22, 2024 18:53

rvu1024 approved these changes May 24, 2024

View reviewed changes

uzhastik approved these changes May 24, 2024

View reviewed changes

alephonea merged commit e830a11 into ydb-platform:main May 24, 2024
3 of 5 checks passed

This was referenced May 27, 2024

Cli changes for autoscaling #4871

Open

Fix getting mid of range for autoscaling #4884

Closed

Fix getting mid range #4885

Merged

Autoscaling logs #4976

Merged

This was referenced May 29, 2024

YQ-3292 fixed kqprun ResourceBrokerConfig passing #4994

Merged

Added ExecutionId for script execution logs #5014

Merged

niksaveliev mentioned this pull request May 30, 2024

Forbid autoscaling with size retention #5054

Merged

		TJoinColumn(TString relName, TString attributeName) : RelName(relName),
		AttributeName(attributeName) {}

Refactoring to perform cardinality estimation specifically for YT. #4482

Refactoring to perform cardinality estimation specifically for YT. #4482

Conversation

alephonea commented May 13, 2024

github-actions bot commented May 13, 2024 • edited

github-actions bot commented May 13, 2024 • edited

github-actions bot commented May 13, 2024 • edited

github-actions bot commented May 14, 2024 • edited

github-actions bot commented May 14, 2024 • edited

github-actions bot commented May 14, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented May 15, 2024 • edited

github-actions bot commented May 15, 2024 • edited

github-actions bot commented May 15, 2024 • edited

github-actions bot commented May 21, 2024 • edited

github-actions bot commented May 21, 2024 • edited

github-actions bot commented May 21, 2024 • edited

github-actions bot commented May 22, 2024 • edited

github-actions bot commented May 22, 2024 • edited

github-actions bot commented May 22, 2024 • edited

github-actions bot commented May 22, 2024 • edited

github-actions bot commented May 22, 2024 • edited

github-actions bot commented May 22, 2024 • edited

github-actions bot commented May 13, 2024 •

edited

github-actions bot commented May 13, 2024 •

edited

github-actions bot commented May 13, 2024 •

edited

github-actions bot commented May 14, 2024 •

edited

github-actions bot commented May 14, 2024 •

edited

github-actions bot commented May 14, 2024 •

edited

github-actions bot commented May 15, 2024 •

edited

github-actions bot commented May 15, 2024 •

edited

github-actions bot commented May 15, 2024 •

edited

github-actions bot commented May 21, 2024 •

edited

github-actions bot commented May 21, 2024 •

edited

github-actions bot commented May 21, 2024 •

edited

github-actions bot commented May 22, 2024 •

edited

github-actions bot commented May 22, 2024 •

edited

github-actions bot commented May 22, 2024 •

edited

github-actions bot commented May 22, 2024 •

edited

github-actions bot commented May 22, 2024 •

edited

github-actions bot commented May 22, 2024 •

edited