Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[improve](txn insert) Txn load support cloud mode #34721

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

mymeiyi
Copy link
Contributor

@mymeiyi mymeiyi commented May 11, 2024

Proposed changes

As the title and leave a todo: Cloud does not support mow table in txn load.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@mymeiyi
Copy link
Contributor Author

mymeiyi commented May 11, 2024

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40555 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit eba8dc6c2738ac6b0fed6f0650c7daeaba952b52, data reload: false

------ Round 1 ----------------------------------
q1	17654	5138	4265	4265
q2	2022	216	190	190
q3	10541	1218	1169	1169
q4	10159	786	878	786
q5	7484	2797	2672	2672
q6	219	129	130	129
q7	1017	562	569	562
q8	9224	2126	2112	2112
q9	9017	6578	6525	6525
q10	8974	3702	3727	3702
q11	478	246	232	232
q12	405	217	208	208
q13	17763	2922	2942	2922
q14	261	217	223	217
q15	514	479	472	472
q16	504	393	387	387
q17	984	694	758	694
q18	8100	7511	7345	7345
q19	6004	1551	1533	1533
q20	660	322	310	310
q21	5128	3837	4055	3837
q22	369	287	286	286
Total cold run time: 117481 ms
Total hot run time: 40555 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4357	4292	4210	4210
q2	377	269	277	269
q3	2982	2784	2730	2730
q4	1874	1657	1623	1623
q5	5325	5310	5317	5310
q6	209	122	122	122
q7	2249	1854	1886	1854
q8	3219	3372	3400	3372
q9	8337	8405	8388	8388
q10	3872	3707	3685	3685
q11	588	492	484	484
q12	780	587	577	577
q13	16483	3010	2982	2982
q14	291	261	263	261
q15	541	484	482	482
q16	470	408	409	408
q17	1781	1477	1515	1477
q18	7731	7516	7490	7490
q19	1745	1611	1584	1584
q20	1968	1766	1737	1737
q21	8049	4938	4991	4938
q22	579	516	495	495
Total cold run time: 73807 ms
Total hot run time: 54478 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185951 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit eba8dc6c2738ac6b0fed6f0650c7daeaba952b52, data reload: false

query1	928	367	341	341
query2	6456	2414	2326	2326
query3	6652	212	208	208
query4	23170	21248	21321	21248
query5	4201	413	420	413
query6	258	174	177	174
query7	4580	286	285	285
query8	239	186	185	185
query9	8719	2442	2407	2407
query10	447	257	243	243
query11	14703	14129	14258	14129
query12	136	88	82	82
query13	1633	367	364	364
query14	10052	6821	7692	6821
query15	242	169	170	169
query16	8003	266	262	262
query17	1853	551	538	538
query18	2023	273	270	270
query19	197	151	146	146
query20	91	84	85	84
query21	193	122	121	121
query22	5085	4839	4832	4832
query23	34256	33757	33231	33231
query24	12040	2826	2835	2826
query25	696	353	362	353
query26	1800	154	152	152
query27	3109	324	319	319
query28	7585	2050	2043	2043
query29	1069	619	592	592
query30	291	151	152	151
query31	967	747	755	747
query32	89	53	53	53
query33	753	265	241	241
query34	1062	489	484	484
query35	844	687	669	669
query36	1044	930	926	926
query37	265	65	67	65
query38	2886	2788	2774	2774
query39	1612	1584	1553	1553
query40	275	128	121	121
query41	40	38	38	38
query42	103	94	96	94
query43	609	541	542	541
query44	1212	711	726	711
query45	263	247	252	247
query46	1100	701	717	701
query47	1969	1849	1879	1849
query48	379	298	288	288
query49	1196	382	382	382
query50	775	386	389	386
query51	6901	6781	6842	6781
query52	105	90	90	90
query53	363	273	282	273
query54	921	431	424	424
query55	79	75	74	74
query56	236	212	213	212
query57	1240	1160	1175	1160
query58	216	190	196	190
query59	3604	3152	3236	3152
query60	259	223	235	223
query61	119	84	87	84
query62	671	464	459	459
query63	301	275	280	275
query64	9718	7379	7382	7379
query65	3146	3104	3092	3092
query66	1392	353	348	348
query67	15437	15079	15180	15079
query68	4513	530	534	530
query69	473	299	301	299
query70	1155	1105	1146	1105
query71	394	269	263	263
query72	7319	2576	2326	2326
query73	711	327	323	323
query74	6587	6208	6085	6085
query75	3503	2661	2638	2638
query76	2753	1006	987	987
query77	401	260	259	259
query78	10633	10153	10255	10153
query79	1391	507	501	501
query80	970	446	430	430
query81	499	223	221	221
query82	1286	96	87	87
query83	238	166	161	161
query84	241	86	83	83
query85	1197	266	263	263
query86	374	301	325	301
query87	3265	3086	3074	3074
query88	3088	2392	2375	2375
query89	477	368	384	368
query90	1902	178	188	178
query91	122	98	95	95
query92	54	53	47	47
query93	1004	515	500	500
query94	1237	182	177	177
query95	392	302	295	295
query96	570	268	263	263
query97	3191	2967	2984	2967
query98	232	215	213	213
query99	1133	904	907	904
Total cold run time: 284163 ms
Total hot run time: 185951 ms

@mymeiyi mymeiyi force-pushed the txn-insert-cloud-2 branch 2 times, most recently from 8078ba3 to 296f6a1 Compare May 13, 2024 08:11
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@mymeiyi
Copy link
Contributor Author

mymeiyi commented May 13, 2024

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 41166 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 296f6a1c089ff50a42286545b8cce34f7b717159, data reload: false

------ Round 1 ----------------------------------
q1	7757	4334	4293	4293
q2	528	198	204	198
q3	1372	1125	1146	1125
q4	1018	761	730	730
q5	2640	2677	2666	2666
q6	220	133	136	133
q7	1015	599	559	559
q8	1997	2367	2166	2166
q9	7064	6836	6775	6775
q10	4045	3909	4031	3909
q11	384	237	241	237
q12	403	219	227	219
q13	16538	3034	3108	3034
q14	282	226	220	220
q15	538	470	460	460
q16	481	411	398	398
q17	947	744	712	712
q18	8473	7877	7889	7877
q19	1615	1579	1520	1520
q20	777	321	335	321
q21	7814	3327	4211	3327
q22	374	288	287	287
Total cold run time: 66282 ms
Total hot run time: 41166 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4419	4374	4344	4344
q2	382	282	268	268
q3	3125	2877	2790	2790
q4	1921	1584	1609	1584
q5	5309	5321	5307	5307
q6	208	124	125	124
q7	2260	1872	1869	1869
q8	3181	3369	3339	3339
q9	8346	8408	8410	8408
q10	3850	3698	3725	3698
q11	582	481	482	481
q12	802	575	608	575
q13	3532	2983	2990	2983
q14	290	262	259	259
q15	517	473	467	467
q16	472	418	416	416
q17	1737	1461	1487	1461
q18	7736	7472	7516	7472
q19	1736	1603	1507	1507
q20	1959	1774	1754	1754
q21	5045	4850	5009	4850
q22	566	504	501	501
Total cold run time: 57975 ms
Total hot run time: 54457 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 188753 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 296f6a1c089ff50a42286545b8cce34f7b717159, data reload: false

query1	914	369	362	362
query2	6410	2374	2332	2332
query3	6644	205	207	205
query4	22886	21278	21198	21198
query5	4113	419	423	419
query6	268	174	172	172
query7	4590	305	289	289
query8	240	183	184	183
query9	8520	2471	2424	2424
query10	454	255	285	255
query11	14742	14159	14156	14156
query12	136	90	93	90
query13	1640	388	394	388
query14	9252	8573	8583	8573
query15	284	172	167	167
query16	7953	257	263	257
query17	1719	568	542	542
query18	2039	291	265	265
query19	195	146	151	146
query20	91	83	86	83
query21	201	128	130	128
query22	5089	4832	4896	4832
query23	34129	33528	33742	33528
query24	6322	2885	2877	2877
query25	532	359	363	359
query26	692	150	149	149
query27	1931	315	321	315
query28	3586	2065	2057	2057
query29	833	598	608	598
query30	240	153	156	153
query31	955	746	745	745
query32	95	52	53	52
query33	481	255	245	245
query34	886	493	485	485
query35	755	675	669	669
query36	1076	941	972	941
query37	110	66	70	66
query38	2835	2754	2761	2754
query39	1643	1569	1569	1569
query40	195	129	123	123
query41	43	38	39	38
query42	104	95	97	95
query43	572	535	554	535
query44	1089	724	740	724
query45	267	252	223	223
query46	1074	746	710	710
query47	1953	1870	1904	1870
query48	385	299	300	299
query49	768	411	396	396
query50	776	386	407	386
query51	6783	6669	6684	6669
query52	104	88	87	87
query53	357	286	296	286
query54	533	466	428	428
query55	73	70	72	70
query56	238	221	221	221
query57	1251	1169	1147	1147
query58	223	201	201	201
query59	3569	3218	3253	3218
query60	257	236	232	232
query61	88	86	87	86
query62	576	476	484	476
query63	306	277	284	277
query64	8454	7514	7478	7478
query65	3109	3105	3101	3101
query66	814	350	347	347
query67	15500	15062	15367	15062
query68	4555	535	537	535
query69	483	310	308	308
query70	1147	1143	1144	1143
query71	383	278	275	275
query72	8099	2684	2522	2522
query73	704	330	326	326
query74	6435	6176	6058	6058
query75	3280	2702	2584	2584
query76	2316	1068	1008	1008
query77	453	267	267	267
query78	10704	10354	10162	10162
query79	4167	511	510	510
query80	986	444	443	443
query81	468	219	221	219
query82	749	99	98	98
query83	204	168	167	167
query84	269	92	85	85
query85	970	268	281	268
query86	412	299	302	299
query87	3252	3109	3135	3109
query88	4349	2473	2476	2473
query89	496	393	402	393
query90	2202	195	194	194
query91	122	96	96	96
query92	64	47	47	47
query93	5118	504	504	504
query94	1061	184	188	184
query95	404	307	308	307
query96	610	281	273	273
query97	3151	2973	3011	2973
query98	240	225	218	218
query99	1210	899	892	892
Total cold run time: 274005 ms
Total hot run time: 188753 ms

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

@@ -1332,6 +1332,317 @@ TEST(MetaServiceTest, CommitTxnExpiredTest) {
}
}

TEST(MetaServiceTest, CommitTxnWithSubTxnTest) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function 'TEST' exceeds recommended size/complexity thresholds [readability-function-size]

TEST(MetaServiceTest, CommitTxnWithSubTxnTest) {
^
Additional context

cloud/test/meta_service_test.cpp:1334: 187 lines including whitespace and comments (threshold 80)

TEST(MetaServiceTest, CommitTxnWithSubTxnTest) {
^

}
}

TEST(MetaServiceTest, BeginAndAbortSubTxnTest) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function 'TEST' exceeds recommended size/complexity thresholds [readability-function-size]

TEST(MetaServiceTest, BeginAndAbortSubTxnTest) {
^
Additional context

cloud/test/meta_service_test.cpp:1523: 120 lines including whitespace and comments (threshold 80)

TEST(MetaServiceTest, BeginAndAbortSubTxnTest) {
^

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

@@ -1332,6 +1332,313 @@ TEST(MetaServiceTest, CommitTxnExpiredTest) {
}
}

TEST(MetaServiceTest, CommitTxnWithSubTxnTest) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function 'TEST' exceeds recommended size/complexity thresholds [readability-function-size]

TEST(MetaServiceTest, CommitTxnWithSubTxnTest) {
^
Additional context

cloud/test/meta_service_test.cpp:1334: 185 lines including whitespace and comments (threshold 80)

TEST(MetaServiceTest, CommitTxnWithSubTxnTest) {
^

}
}

TEST(MetaServiceTest, BeginAndAbortSubTxnTest) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function 'TEST' exceeds recommended size/complexity thresholds [readability-function-size]

TEST(MetaServiceTest, BeginAndAbortSubTxnTest) {
^
Additional context

cloud/test/meta_service_test.cpp:1521: 118 lines including whitespace and comments (threshold 80)

TEST(MetaServiceTest, BeginAndAbortSubTxnTest) {
^

@mymeiyi
Copy link
Contributor Author

mymeiyi commented May 15, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41841 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a493ddf78ca0cc05b6b7240a591566de30184d4b, data reload: false

------ Round 1 ----------------------------------
q1	17614	4270	4228	4228
q2	2020	191	198	191
q3	10438	1242	1259	1242
q4	10199	804	823	804
q5	7477	2741	2650	2650
q6	226	132	132	132
q7	1020	618	585	585
q8	9214	2185	2112	2112
q9	9282	6742	6688	6688
q10	9718	3951	3847	3847
q11	463	240	240	240
q12	445	220	232	220
q13	17204	3120	3269	3120
q14	257	226	228	226
q15	532	462	479	462
q16	477	374	366	366
q17	983	652	688	652
q18	8473	8159	7816	7816
q19	7204	1550	1571	1550
q20	637	322	318	318
q21	5121	4113	4295	4113
q22	371	281	279	279
Total cold run time: 119375 ms
Total hot run time: 41841 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4501	4408	4391	4391
q2	385	288	277	277
q3	3131	2955	2797	2797
q4	1913	1595	1657	1595
q5	5483	5459	5500	5459
q6	221	127	129	127
q7	2354	1965	1973	1965
q8	3278	3394	3421	3394
q9	8686	8707	8611	8611
q10	3983	3852	3834	3834
q11	581	487	502	487
q12	800	579	607	579
q13	16764	3148	3222	3148
q14	311	264	264	264
q15	511	483	459	459
q16	477	423	406	406
q17	1771	1489	1478	1478
q18	7718	7704	7429	7429
q19	1680	1551	1531	1531
q20	1970	1785	1763	1763
q21	4917	4798	4917	4798
q22	605	495	502	495
Total cold run time: 72040 ms
Total hot run time: 55287 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 187484 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a493ddf78ca0cc05b6b7240a591566de30184d4b, data reload: false

query1	917	393	376	376
query2	6458	2491	2324	2324
query3	6627	211	209	209
query4	22790	21239	21234	21234
query5	4127	423	448	423
query6	265	193	170	170
query7	4583	311	292	292
query8	239	185	195	185
query9	8464	2437	2400	2400
query10	435	258	264	258
query11	14764	14271	14156	14156
query12	136	93	90	90
query13	1654	377	381	377
query14	10548	7365	8387	7365
query15	256	163	171	163
query16	8100	259	266	259
query17	1733	567	544	544
query18	2110	282	269	269
query19	203	153	149	149
query20	93	89	84	84
query21	199	131	130	130
query22	5035	4862	4918	4862
query23	34074	33518	33740	33518
query24	6916	2928	2908	2908
query25	561	373	373	373
query26	691	159	153	153
query27	2176	321	331	321
query28	4979	2091	2054	2054
query29	832	613	605	605
query30	275	176	182	176
query31	995	779	751	751
query32	90	51	52	51
query33	514	248	257	248
query34	870	485	478	478
query35	766	688	677	677
query36	1068	891	898	891
query37	113	68	69	68
query38	2874	2759	2787	2759
query39	1606	1548	1611	1548
query40	198	128	122	122
query41	50	44	43	43
query42	100	94	96	94
query43	560	545	530	530
query44	1092	726	740	726
query45	267	256	233	233
query46	1070	710	729	710
query47	1970	1898	1920	1898
query48	375	295	292	292
query49	863	398	402	398
query50	786	392	414	392
query51	6938	6868	6730	6730
query52	108	93	92	92
query53	351	286	292	286
query54	541	440	433	433
query55	75	75	75	75
query56	255	231	233	231
query57	1217	1148	1189	1148
query58	228	210	208	208
query59	3289	3102	3091	3091
query60	293	253	251	251
query61	111	108	105	105
query62	625	489	479	479
query63	310	297	299	297
query64	8541	7475	7511	7475
query65	3140	3080	3106	3080
query66	791	373	354	354
query67	15456	15194	15114	15114
query68	6534	559	554	554
query69	546	312	324	312
query70	1215	1143	1072	1072
query71	458	275	280	275
query72	7869	2693	2493	2493
query73	719	340	333	333
query74	6525	6168	6059	6059
query75	3853	2680	2580	2580
query76	3853	1070	957	957
query77	735	274	276	274
query78	10746	10065	10116	10065
query79	6073	522	517	517
query80	1107	471	454	454
query81	515	386	242	242
query82	894	99	96	96
query83	201	165	167	165
query84	271	82	89	82
query85	1222	269	272	269
query86	446	327	323	323
query87	3295	3101	3077	3077
query88	4980	2437	2434	2434
query89	498	388	396	388
query90	2014	188	193	188
query91	127	104	101	101
query92	61	50	49	49
query93	5624	529	505	505
query94	1191	183	182	182
query95	410	306	309	306
query96	609	269	272	269
query97	3128	3011	2967	2967
query98	233	223	209	209
query99	1213	907	902	902
Total cold run time: 285901 ms
Total hot run time: 187484 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.74 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a493ddf78ca0cc05b6b7240a591566de30184d4b, data reload: false

query1	0.05	0.04	0.03
query2	0.08	0.04	0.03
query3	0.23	0.05	0.06
query4	1.66	0.07	0.09
query5	0.50	0.48	0.50
query6	1.13	0.73	0.73
query7	0.02	0.01	0.02
query8	0.05	0.04	0.05
query9	0.53	0.50	0.47
query10	0.54	0.55	0.53
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.60	0.59	0.61
query14	0.78	0.77	0.77
query15	0.83	0.81	0.80
query16	0.36	0.36	0.36
query17	1.02	1.01	0.95
query18	0.20	0.25	0.25
query19	1.82	1.72	1.66
query20	0.02	0.01	0.01
query21	15.44	0.75	0.69
query22	4.70	6.67	2.29
query23	18.28	1.33	1.26
query24	1.57	0.32	0.23
query25	0.16	0.08	0.08
query26	0.26	0.16	0.16
query27	0.08	0.08	0.08
query28	13.34	1.01	1.00
query29	13.22	3.31	3.26
query30	0.24	0.06	0.05
query31	2.85	0.38	0.39
query32	3.29	0.47	0.46
query33	2.80	2.86	2.79
query34	17.12	4.42	4.46
query35	4.47	4.51	4.66
query36	0.64	0.45	0.47
query37	0.17	0.15	0.15
query38	0.15	0.15	0.15
query39	0.04	0.04	0.03
query40	0.17	0.13	0.15
query41	0.09	0.05	0.04
query42	0.06	0.04	0.05
query43	0.04	0.04	0.03
Total cold run time: 109.91 s
Total hot run time: 30.74 s

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

}
}

TEST(MetaServiceTest, BeginAndAbortSubTxnTest) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function 'TEST' exceeds recommended size/complexity thresholds [readability-function-size]

TEST(MetaServiceTest, BeginAndAbortSubTxnTest) {
^
Additional context

cloud/test/meta_service_test.cpp:1521: 119 lines including whitespace and comments (threshold 80)

TEST(MetaServiceTest, BeginAndAbortSubTxnTest) {
^

@mymeiyi
Copy link
Contributor Author

mymeiyi commented May 15, 2024

run buildall

@mymeiyi
Copy link
Contributor Author

mymeiyi commented May 20, 2024

run buildall

@mymeiyi
Copy link
Contributor Author

mymeiyi commented May 22, 2024

run buildall

@doris-robot
Copy link

TPC-DS: Total hot run time: 182574 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 16a2691ebf2cf85daa731860eff5651d5a6e11e4, data reload: false

query1	909	387	371	371
query2	6259	2406	2314	2314
query3	6648	207	210	207
query4	24678	21772	21881	21772
query5	3938	441	427	427
query6	266	177	173	173
query7	4668	308	305	305
query8	241	187	183	183
query9	8650	2465	2493	2465
query10	426	281	248	248
query11	15348	14797	14948	14797
query12	123	85	98	85
query13	1595	379	388	379
query14	10976	8374	6869	6869
query15	245	165	171	165
query16	7596	260	248	248
query17	1766	547	535	535
query18	1957	273	265	265
query19	319	146	151	146
query20	85	90	81	81
query21	190	132	126	126
query22	5106	4914	4805	4805
query23	34050	33642	33568	33568
query24	10720	2912	2892	2892
query25	603	356	364	356
query26	1133	154	155	154
query27	2349	315	310	310
query28	7250	2034	2041	2034
query29	859	610	619	610
query30	270	171	173	171
query31	923	736	742	736
query32	92	49	50	49
query33	754	242	244	242
query34	1038	489	477	477
query35	855	668	649	649
query36	1069	915	928	915
query37	126	73	91	73
query38	2919	2780	2770	2770
query39	1599	1550	1691	1550
query40	196	123	123	123
query41	46	44	44	44
query42	100	93	95	93
query43	566	530	518	518
query44	1223	737	742	737
query45	270	255	247	247
query46	1081	723	719	719
query47	1965	1887	1893	1887
query48	376	299	297	297
query49	886	401	396	396
query50	763	380	387	380
query51	6818	6868	6779	6779
query52	102	94	92	92
query53	353	277	282	277
query54	937	425	434	425
query55	73	75	71	71
query56	250	221	230	221
query57	1241	1188	1151	1151
query58	226	204	194	194
query59	3591	3039	3051	3039
query60	261	235	241	235
query61	91	88	104	88
query62	682	472	494	472
query63	308	281	281	281
query64	8995	2240	1782	1782
query65	3150	3115	3103	3103
query66	858	340	329	329
query67	15626	15174	15145	15145
query68	4528	552	548	548
query69	478	299	302	299
query70	1159	1141	1135	1135
query71	426	275	273	273
query72	7255	2565	2380	2380
query73	717	318	320	318
query74	6710	6246	6263	6246
query75	3409	2631	2632	2631
query76	2705	1008	955	955
query77	372	286	267	267
query78	10552	10491	10155	10155
query79	2138	520	517	517
query80	1214	464	513	464
query81	547	253	245	245
query82	742	95	97	95
query83	258	169	167	167
query84	241	84	82	82
query85	1479	279	275	275
query86	480	320	282	282
query87	3391	3185	3128	3128
query88	3942	2366	2383	2366
query89	476	383	386	383
query90	1974	196	189	189
query91	126	104	97	97
query92	64	50	50	50
query93	1767	517	510	510
query94	1179	186	183	183
query95	394	308	308	308
query96	649	267	258	258
query97	3221	3006	3013	3006
query98	291	225	209	209
query99	1222	947	906	906
Total cold run time: 283738 ms
Total hot run time: 182574 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.28 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 16a2691ebf2cf85daa731860eff5651d5a6e11e4, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.06	0.06
query4	1.65	0.09	0.11
query5	0.51	0.50	0.50
query6	1.12	0.72	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.05
query9	0.54	0.48	0.49
query10	0.54	0.56	0.55
query11	0.17	0.12	0.11
query12	0.15	0.13	0.12
query13	0.60	0.59	0.59
query14	0.79	0.80	0.78
query15	0.86	0.81	0.81
query16	0.37	0.35	0.37
query17	0.99	1.01	1.00
query18	0.20	0.27	0.22
query19	1.81	1.68	1.68
query20	0.02	0.01	0.01
query21	15.75	0.66	0.65
query22	4.18	7.53	1.54
query23	18.28	1.43	1.35
query24	2.10	0.22	0.21
query25	0.16	0.08	0.08
query26	0.26	0.17	0.16
query27	0.08	0.09	0.08
query28	13.22	1.03	1.01
query29	13.82	3.30	3.26
query30	0.24	0.06	0.05
query31	2.87	0.39	0.38
query32	3.28	0.48	0.47
query33	2.82	2.90	2.91
query34	17.20	4.47	4.44
query35	4.51	4.47	4.62
query36	0.65	0.46	0.46
query37	0.18	0.15	0.15
query38	0.15	0.14	0.15
query39	0.04	0.03	0.04
query40	0.17	0.17	0.14
query41	0.09	0.04	0.04
query42	0.05	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 110.88 s
Total hot run time: 30.28 s

@mymeiyi
Copy link
Contributor Author

mymeiyi commented May 22, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40766 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7b99d1712330a09e75ab606100b0965aa8a8877d, data reload: false

------ Round 1 ----------------------------------
q1	17647	4368	4336	4336
q2	2028	197	196	196
q3	10486	1272	1176	1176
q4	10501	735	870	735
q5	7868	2698	2657	2657
q6	225	136	135	135
q7	1024	612	583	583
q8	9874	2133	2115	2115
q9	9814	7240	7180	7180
q10	9273	3735	3708	3708
q11	443	248	235	235
q12	453	217	227	217
q13	18363	2980	3003	2980
q14	267	215	216	215
q15	512	485	482	482
q16	525	389	378	378
q17	976	752	736	736
q18	8017	7419	7440	7419
q19	1653	1484	1456	1456
q20	654	302	295	295
q21	5070	3254	4042	3254
q22	351	278	292	278
Total cold run time: 116024 ms
Total hot run time: 40766 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4348	4263	4251	4251
q2	370	285	265	265
q3	3039	2768	2766	2766
q4	1894	1636	1605	1605
q5	5268	5277	5296	5277
q6	213	126	129	126
q7	2225	1891	1872	1872
q8	3202	3364	3355	3355
q9	8948	9035	9017	9017
q10	3914	3728	3713	3713
q11	582	497	495	495
q12	757	587	574	574
q13	17302	2985	3016	2985
q14	302	266	255	255
q15	522	469	465	465
q16	469	418	424	418
q17	1775	1485	1495	1485
q18	7598	7727	7453	7453
q19	5616	1607	1552	1552
q20	2033	1793	1786	1786
q21	5026	4935	4830	4830
q22	566	497	503	497
Total cold run time: 75969 ms
Total hot run time: 55042 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 180656 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7b99d1712330a09e75ab606100b0965aa8a8877d, data reload: false

query1	909	387	363	363
query2	6461	2411	2371	2371
query3	6648	211	215	211
query4	23132	21381	21219	21219
query5	4140	424	412	412
query6	256	175	173	173
query7	4588	288	281	281
query8	239	185	197	185
query9	8598	2384	2373	2373
query10	441	252	248	248
query11	15102	14201	14134	14134
query12	139	92	90	90
query13	1645	368	359	359
query14	10488	8409	7477	7477
query15	255	165	170	165
query16	8156	254	254	254
query17	1865	556	582	556
query18	2110	272	265	265
query19	205	144	147	144
query20	93	89	88	88
query21	199	130	125	125
query22	4996	4868	4775	4775
query23	34401	33223	33520	33223
query24	11833	2937	2904	2904
query25	654	365	360	360
query26	1731	156	153	153
query27	2950	311	326	311
query28	7570	2027	2016	2016
query29	1008	627	592	592
query30	284	175	178	175
query31	998	752	738	738
query32	96	53	55	53
query33	748	244	265	244
query34	1059	469	478	469
query35	800	672	660	660
query36	1086	894	912	894
query37	193	71	69	69
query38	2872	2755	2749	2749
query39	1622	1576	1558	1558
query40	273	126	122	122
query41	46	45	44	44
query42	106	94	97	94
query43	587	537	535	535
query44	1263	729	728	728
query45	270	251	244	244
query46	1075	709	727	709
query47	1957	1888	1891	1888
query48	369	287	289	287
query49	1203	389	407	389
query50	760	402	378	378
query51	6839	6778	6745	6745
query52	104	87	90	87
query53	346	283	280	280
query54	1023	420	427	420
query55	73	73	72	72
query56	279	226	227	226
query57	1263	1158	1182	1158
query58	230	201	202	201
query59	3672	3296	3168	3168
query60	252	233	231	231
query61	94	89	89	89
query62	662	466	466	466
query63	316	278	280	278
query64	9824	2239	1740	1740
query65	3250	3155	3112	3112
query66	1381	359	331	331
query67	15295	14933	15082	14933
query68	4533	523	513	513
query69	460	303	296	296
query70	1175	1086	1132	1086
query71	370	269	282	269
query72	7334	2574	2399	2399
query73	706	322	321	321
query74	6630	6188	6237	6188
query75	3402	2639	2579	2579
query76	2944	945	959	945
query77	415	263	265	263
query78	10848	10092	9896	9896
query79	2156	515	511	511
query80	985	441	428	428
query81	521	247	245	245
query82	633	95	96	95
query83	237	163	160	160
query84	236	87	83	83
query85	1848	329	260	260
query86	475	321	307	307
query87	3256	3123	3068	3068
query88	4202	2308	2322	2308
query89	472	388	385	385
query90	2017	185	183	183
query91	120	99	98	98
query92	62	47	47	47
query93	2246	506	490	490
query94	1221	175	190	175
query95	386	291	289	289
query96	581	258	263	258
query97	3172	3010	3009	3009
query98	236	218	217	217
query99	1140	894	921	894
Total cold run time: 288401 ms
Total hot run time: 180656 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.21 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7b99d1712330a09e75ab606100b0965aa8a8877d, data reload: false

query1	0.04	0.04	0.03
query2	0.09	0.04	0.04
query3	0.23	0.05	0.05
query4	1.66	0.08	0.09
query5	0.49	0.49	0.49
query6	1.13	0.73	0.72
query7	0.03	0.01	0.01
query8	0.05	0.05	0.04
query9	0.53	0.48	0.49
query10	0.54	0.55	0.54
query11	0.16	0.11	0.12
query12	0.15	0.12	0.12
query13	0.61	0.58	0.62
query14	0.79	0.76	0.78
query15	0.83	0.81	0.80
query16	0.36	0.37	0.37
query17	1.03	0.95	1.01
query18	0.22	0.23	0.25
query19	1.84	1.75	1.78
query20	0.01	0.01	0.02
query21	15.43	0.69	0.68
query22	4.53	6.24	2.58
query23	18.31	1.29	1.29
query24	1.45	0.41	0.22
query25	0.15	0.08	0.08
query26	0.27	0.17	0.18
query27	0.08	0.08	0.07
query28	13.23	1.02	1.00
query29	12.67	3.27	3.27
query30	0.24	0.06	0.05
query31	2.88	0.38	0.38
query32	3.29	0.46	0.48
query33	2.93	2.88	2.87
query34	17.37	4.44	4.48
query35	4.49	4.51	4.45
query36	0.65	0.46	0.45
query37	0.17	0.15	0.16
query38	0.17	0.15	0.15
query39	0.05	0.03	0.03
query40	0.17	0.16	0.14
query41	0.09	0.04	0.04
query42	0.05	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.5 s
Total hot run time: 31.21 s

dataroaring
dataroaring previously approved these changes May 22, 2024
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 22, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@mymeiyi
Copy link
Contributor Author

mymeiyi commented May 22, 2024

run cloud_p0 cloud_ut

@mymeiyi
Copy link
Contributor Author

mymeiyi commented May 22, 2024

run cloud_p0

@mymeiyi
Copy link
Contributor Author

mymeiyi commented May 22, 2024

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label May 22, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 41995 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0269b4ba2bd3e692e602f3bd1497ff6c0e1a6e86, data reload: false

------ Round 1 ----------------------------------
q1	17608	4317	4253	4253
q2	2017	191	192	191
q3	10463	1248	1208	1208
q4	10198	753	818	753
q5	7523	2749	2690	2690
q6	218	135	137	135
q7	971	607	625	607
q8	9229	2156	2135	2135
q9	9282	6697	6726	6697
q10	9176	3843	3873	3843
q11	461	239	243	239
q12	496	221	219	219
q13	17284	3237	3244	3237
q14	261	215	230	215
q15	534	479	472	472
q16	502	409	404	404
q17	1015	795	686	686
q18	8468	7964	7813	7813
q19	7387	1558	1526	1526
q20	665	325	308	308
q21	5202	4088	4108	4088
q22	355	292	276	276
Total cold run time: 119315 ms
Total hot run time: 41995 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4483	4372	4405	4372
q2	395	270	278	270
q3	3144	2908	2911	2908
q4	1998	1676	1609	1609
q5	5300	5516	5501	5501
q6	222	128	125	125
q7	2194	1818	1834	1818
q8	3248	3424	3420	3420
q9	8634	8621	8760	8621
q10	4027	3749	3699	3699
q11	588	498	480	480
q12	798	619	623	619
q13	17088	3145	3166	3145
q14	316	261	261	261
q15	530	484	494	484
q16	488	423	454	423
q17	1815	1547	1516	1516
q18	7803	7510	7374	7374
q19	9386	1654	1529	1529
q20	2072	1805	1748	1748
q21	10212	4654	4756	4654
q22	574	473	485	473
Total cold run time: 85315 ms
Total hot run time: 55049 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 168882 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0269b4ba2bd3e692e602f3bd1497ff6c0e1a6e86, data reload: false

query1	902	401	369	369
query2	6435	2385	2276	2276
query3	6857	210	214	210
query4	19572	17242	17253	17242
query5	4138	425	424	424
query6	246	158	176	158
query7	4582	302	287	287
query8	233	188	190	188
query9	8563	2392	2356	2356
query10	453	306	274	274
query11	10593	10181	10061	10061
query12	135	88	95	88
query13	1658	370	361	361
query14	9201	6816	7133	6816
query15	226	171	175	171
query16	7077	283	260	260
query17	1001	512	534	512
query18	1798	273	266	266
query19	200	157	157	157
query20	92	85	86	85
query21	197	126	128	126
query22	4336	4167	4003	4003
query23	33696	32593	33201	32593
query24	5041	2866	2790	2790
query25	472	373	365	365
query26	690	158	163	158
query27	1841	332	324	324
query28	3527	2057	2074	2057
query29	841	616	602	602
query30	249	172	175	172
query31	918	758	736	736
query32	60	52	53	52
query33	441	268	262	262
query34	846	474	499	474
query35	714	642	599	599
query36	1044	893	914	893
query37	108	71	68	68
query38	2884	2722	2769	2722
query39	841	783	774	774
query40	201	124	123	123
query41	45	44	43	43
query42	104	95	99	95
query43	607	542	559	542
query44	1073	726	746	726
query45	180	171	162	162
query46	1060	701	728	701
query47	1845	1775	1765	1765
query48	369	302	311	302
query49	762	376	379	376
query50	761	383	389	383
query51	6932	6836	6737	6737
query52	99	94	93	93
query53	345	300	289	289
query54	558	428	433	428
query55	74	74	74	74
query56	261	237	241	237
query57	1097	1066	1056	1056
query58	222	207	212	207
query59	3515	3323	3160	3160
query60	284	258	253	253
query61	106	86	84	84
query62	563	471	456	456
query63	313	289	290	289
query64	2605	1730	1695	1695
query65	3178	3108	3107	3107
query66	731	328	321	321
query67	15374	14972	14795	14795
query68	4522	526	536	526
query69	431	268	267	267
query70	1180	1155	1198	1155
query71	397	267	269	267
query72	7570	2693	2596	2596
query73	702	325	325	325
query74	6017	5606	5587	5587
query75	3312	2621	2653	2621
query76	2239	957	974	957
query77	392	274	271	271
query78	10426	9816	9736	9736
query79	2412	514	517	514
query80	1056	456	440	440
query81	516	240	242	240
query82	757	102	96	96
query83	254	177	188	177
query84	265	85	84	84
query85	1137	322	315	315
query86	462	290	285	285
query87	3259	3131	3140	3131
query88	4109	2347	2351	2347
query89	460	396	378	378
query90	2065	196	193	193
query91	134	109	106	106
query92	68	52	51	51
query93	2222	516	504	504
query94	1259	195	200	195
query95	419	344	337	337
query96	587	275	269	269
query97	3193	3040	3016	3016
query98	249	220	222	220
query99	1222	846	866	846
Total cold run time: 249265 ms
Total hot run time: 168882 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.79 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0269b4ba2bd3e692e602f3bd1497ff6c0e1a6e86, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.07	0.06
query4	1.66	0.09	0.09
query5	0.53	0.49	0.50
query6	1.13	0.72	0.73
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.54	0.49	0.48
query10	0.55	0.54	0.54
query11	0.15	0.11	0.11
query12	0.14	0.12	0.11
query13	0.59	0.59	0.59
query14	0.80	0.78	0.79
query15	0.83	0.81	0.81
query16	0.36	0.36	0.35
query17	0.96	0.98	0.97
query18	0.23	0.22	0.27
query19	1.74	1.68	1.62
query20	0.02	0.01	0.01
query21	15.73	0.65	0.65
query22	4.81	6.96	2.13
query23	18.31	1.41	1.28
query24	1.36	0.32	0.27
query25	0.14	0.08	0.09
query26	0.25	0.16	0.16
query27	0.08	0.08	0.08
query28	13.42	1.01	1.00
query29	13.28	3.31	3.31
query30	0.24	0.05	0.06
query31	2.86	0.39	0.39
query32	3.27	0.47	0.47
query33	2.88	2.92	2.90
query34	17.20	4.43	4.39
query35	4.47	4.48	4.49
query36	0.69	0.48	0.48
query37	0.18	0.16	0.16
query38	0.15	0.14	0.15
query39	0.04	0.04	0.04
query40	0.16	0.14	0.14
query41	0.09	0.05	0.05
query42	0.05	0.05	0.04
query43	0.04	0.03	0.04
Total cold run time: 110.35 s
Total hot run time: 30.79 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants