Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](Nereids) fix sql cache return old value when truncate partition #34698

Merged
merged 1 commit into from May 13, 2024
Merged

[fix](Nereids) fix sql cache return old value when truncate partition #34698

merged 1 commit into from May 13, 2024

Conversation

924060929
Copy link
Contributor

  1. fix sql cache return old value when truncate partition
  2. use expire_sql_cache_in_fe_second to control the expire time of the sql cache which in the NereidsSqlCacheManager

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@924060929
Copy link
Contributor Author

run buildall

@924060929 924060929 force-pushed the fix_sql_cache_check_partition branch from 45f8994 to ae1dca5 Compare May 11, 2024 07:37
@924060929
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41762 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ae1dca5c9b0d45b17838288f4647f5d0fbec7ed2, data reload: false

------ Round 1 ----------------------------------
q1	17691	4670	4239	4239
q2	2014	200	193	193
q3	5986	1227	1406	1227
q4	9946	830	904	830
q5	7546	2758	2775	2758
q6	223	137	139	137
q7	1096	614	635	614
q8	9328	2200	2128	2128
q9	9308	6891	6626	6626
q10	9015	3965	3899	3899
q11	466	240	236	236
q12	444	230	227	227
q13	17074	3141	3159	3141
q14	284	226	240	226
q15	508	484	475	475
q16	550	410	399	399
q17	997	738	784	738
q18	8404	7936	7428	7428
q19	4563	1534	1548	1534
q20	683	332	321	321
q21	5217	4319	4099	4099
q22	351	296	287	287
Total cold run time: 111694 ms
Total hot run time: 41762 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4541	4233	4267	4233
q2	375	258	263	258
q3	2996	2830	2697	2697
q4	1894	1649	1593	1593
q5	5313	5372	5293	5293
q6	213	124	130	124
q7	2280	1849	1856	1849
q8	3168	3332	3423	3332
q9	8406	8440	8408	8408
q10	3930	3785	3669	3669
q11	582	490	491	490
q12	766	566	576	566
q13	16927	3056	2976	2976
q14	287	269	255	255
q15	529	460	468	460
q16	476	418	418	418
q17	1781	1510	1452	1452
q18	7646	7544	7324	7324
q19	1710	1611	1635	1611
q20	1965	1775	1763	1763
q21	4987	4861	4810	4810
q22	596	504	507	504
Total cold run time: 71368 ms
Total hot run time: 54085 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 187802 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ae1dca5c9b0d45b17838288f4647f5d0fbec7ed2, data reload: false

query1	909	361	346	346
query2	6450	2374	2313	2313
query3	6644	218	215	215
query4	25237	21194	21268	21194
query5	4090	417	407	407
query6	269	170	172	170
query7	4582	299	295	295
query8	245	193	200	193
query9	8592	2410	2413	2410
query10	452	255	258	255
query11	14804	14337	14147	14147
query12	133	95	88	88
query13	1648	378	380	378
query14	10867	8316	7870	7870
query15	230	173	177	173
query16	7888	259	269	259
query17	1755	552	532	532
query18	1950	284	275	275
query19	199	152	146	146
query20	90	90	86	86
query21	192	129	125	125
query22	5079	4849	4882	4849
query23	33998	33469	33615	33469
query24	6869	2928	2879	2879
query25	587	366	372	366
query26	715	154	150	150
query27	2124	312	312	312
query28	4471	2090	2064	2064
query29	844	607	588	588
query30	253	158	159	158
query31	950	754	746	746
query32	91	54	54	54
query33	496	246	249	246
query34	870	478	490	478
query35	772	708	714	708
query36	1054	900	912	900
query37	105	69	66	66
query38	2897	2777	2784	2777
query39	1610	1550	1554	1550
query40	193	125	124	124
query41	45	39	38	38
query42	102	96	94	94
query43	591	518	531	518
query44	1084	722	732	722
query45	281	252	260	252
query46	1079	711	696	696
query47	1991	1906	1909	1906
query48	371	295	296	295
query49	877	394	403	394
query50	760	392	380	380
query51	7296	7137	7119	7119
query52	102	98	86	86
query53	348	286	281	281
query54	538	419	424	419
query55	77	73	74	73
query56	278	221	216	216
query57	1218	1177	1137	1137
query58	212	195	202	195
query59	3422	3319	3165	3165
query60	266	236	238	236
query61	88	89	88	88
query62	622	490	471	471
query63	309	281	280	280
query64	8478	7518	7364	7364
query65	3161	3113	3078	3078
query66	792	350	345	345
query67	15392	15165	14816	14816
query68	4468	526	530	526
query69	464	301	321	301
query70	1183	1125	1126	1125
query71	375	273	265	265
query72	7590	2581	2400	2400
query73	712	330	341	330
query74	6450	6201	6096	6096
query75	3298	2640	2639	2639
query76	2308	968	987	968
query77	417	264	261	261
query78	10593	10463	10241	10241
query79	1069	505	534	505
query80	656	445	432	432
query81	511	217	221	217
query82	177	90	95	90
query83	192	167	161	161
query84	251	86	84	84
query85	858	266	266	266
query86	329	307	294	294
query87	3353	3137	3130	3130
query88	3320	2412	2456	2412
query89	475	371	376	371
query90	2092	186	188	186
query91	125	100	98	98
query92	57	51	49	49
query93	1028	524	495	495
query94	1251	180	180	180
query95	407	309	308	308
query96	582	271	266	266
query97	3174	2970	2984	2970
query98	234	222	222	222
query99	1155	897	889	889
Total cold run time: 270138 ms
Total hot run time: 187802 ms

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 11, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

morrySnow
morrySnow approved these changes May 13, 2024
View reviewed changes
fe/fe-common/src/main/java/org/apache/doris/common/Config.java
@@ -1314,12 +1314,18 @@ public class Config extends ConfigBase {
* Minimum interval between last version when caching results,
* This parameter distinguishes between offline and real-time updates
*/
@ConfField(mutable = true, masterOnly = false)
public static int cache_last_version_interval_second = 30;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add description in annotation

fe/fe-common/src/main/java/org/apache/doris/common/Config.java
@ConfField(
mutable = true,
masterOnly = false,
callbackClassString = "org.apache.doris.common.NereidsSqlCacheManager$UpdateConfig"
)
public static int cache_last_version_interval_second = 30;
public static int expire_sql_cache_in_fe_second = 300;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add description in annotation

@morrySnow morrySnow merged commit 4f1a884 into apache:master May 13, 2024
26 of 29 checks passed
@924060929 924060929 deleted the fix_sql_cache_check_partition branch May 13, 2024 04:06
nextdreamblue pushed a commit to nextdreamblue/doris that referenced this pull request May 13, 2024
@924060929 @nextdreamblue
[fix](Nereids) fix sql cache return old value when truncate partition (…
e9211d4 
…apache#34698)

1. fix sql cache return old value when truncate partition
2. use expire_sql_cache_in_fe_second to control the expire time of the sql cache which in the NereidsSqlCacheManager
M1saka2003 pushed a commit to M1saka2003/doris that referenced this pull request May 14, 2024
@924060929 @M1saka2003
[fix](Nereids) fix sql cache return old value when truncate partition (…
5d6a000 
…apache#34698)

1. fix sql cache return old value when truncate partition
2. use expire_sql_cache_in_fe_second to control the expire time of the sql cache which in the NereidsSqlCacheManager
ByteYue pushed a commit to ByteYue/doris that referenced this pull request May 15, 2024
@924060929 @ByteYue
[fix](Nereids) fix sql cache return old value when truncate partition (…
6c82500 
…apache#34698)

1. fix sql cache return old value when truncate partition
2. use expire_sql_cache_in_fe_second to control the expire time of the sql cache which in the NereidsSqlCacheManager
yiguolei pushed a commit that referenced this pull request May 18, 2024
@924060929 @Doris-Extras
[fix](Nereids) fix sql cache return old value when truncate partition (…
5012ddd 
…#34698)

1. fix sql cache return old value when truncate partition
2. use expire_sql_cache_in_fe_second to control the expire time of the sql cache which in the NereidsSqlCacheManager
M1saka2003 pushed a commit to M1saka2003/doris that referenced this pull request May 24, 2024
@924060929 @M1saka2003
[fix](Nereids) fix sql cache return old value when truncate partition (…
54b9670 
…apache#34698)

1. fix sql cache return old value when truncate partition
2. use expire_sql_cache_in_fe_second to control the expire time of the sql cache which in the NereidsSqlCacheManager
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@starocean999 starocean999 starocean999 approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.4-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@924060929 @doris-robot @starocean999 @morrySnow @xiaokang