Skip to content

[fix](filecache) add ttl mgr NOT_FOUND cleanup #60269

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
freemandealer wants to merge 1 commit into
base: master
Choose a base branch
from
Open

[fix](filecache) add ttl mgr NOT_FOUND cleanup #60269

freemandealer wants to merge 1 commit into from
+26 −2

Conversation

@freemandealer
Copy link
Contributor

@freemandealer freemandealer commented Jan 27, 2026

This commit does the followings:

  1. Drop missing tablet ids on NOT_FOUND to stop repeated TTL meta lookups and log spam
  2. add bvar for tablet-id set size to monitor cleanup behavior

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@freemandealer
[fix](filecache) add ttl mgr NOT_FOUND cleanup
4c62152 
This commit does the followings:
1. Drop missing tablet ids on NOT_FOUND to stop repeated TTL meta lookups and log spam
2. add bvar for tablet-id set size to monitor cleanup behavior

Signed-off-by: zhengyu <[email protected]>
@Thearas
Copy link
Contributor

Thearas commented Jan 27, 2026

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@freemandealer
Copy link
Contributor Author

freemandealer commented Jan 27, 2026

run buildall

@doris-robot
Copy link

doris-robot commented Jan 27, 2026

TPC-H: Total hot run time: 32847 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4c62152467a5aebd4a5887e5d964dbb591023a71, data reload: false

------ Round 1 ----------------------------------
q1	17708	5350	5113	5113
q2	1995	330	205	205
q3	10197	1344	786	786
q4	10219	889	329	329
q5	7880	2154	1963	1963
q6	215	186	151	151
q7	883	740	599	599
q8	9272	1386	1087	1087
q9	5413	4796	4853	4796
q10	6836	1914	1573	1573
q11	522	302	271	271
q12	363	377	228	228
q13	17801	4083	3305	3305
q14	247	245	224	224
q15	918	817	819	817
q16	688	676	620	620
q17	649	759	561	561
q18	7340	6509	6600	6509
q19	1275	986	619	619
q20	392	349	232	232
q21	2654	1901	1949	1901
q22	1048	1054	958	958
Total cold run time: 104515 ms
Total hot run time: 32847 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5385	5299	5320	5299
q2	265	346	260	260
q3	2164	2678	2243	2243
q4	1359	1788	1316	1316
q5	4281	4185	4750	4185
q6	248	198	156	156
q7	2147	1999	1863	1863
q8	2591	2433	2446	2433
q9	7747	7534	7372	7372
q10	2873	3117	2608	2608
q11	572	479	460	460
q12	723	758	627	627
q13	3989	4285	3652	3652
q14	298	321	291	291
q15	864	817	819	817
q16	713	777	698	698
q17	1173	1309	1343	1309
q18	8243	7852	7872	7852
q19	926	852	1065	852
q20	2087	2165	2046	2046
q21	4748	4251	4094	4094
q22	1109	1019	945	945
Total cold run time: 54505 ms
Total hot run time: 51378 ms

@doris-robot
Copy link

doris-robot commented Jan 27, 2026

ClickBench: Total hot run time: 28.25 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4c62152467a5aebd4a5887e5d964dbb591023a71, data reload: false

query1	0.05	0.05	0.05
query2	0.10	0.04	0.04
query3	0.26	0.09	0.08
query4	1.62	0.11	0.11
query5	0.26	0.24	0.26
query6	1.16	0.69	0.68
query7	0.03	0.03	0.03
query8	0.05	0.04	0.04
query9	0.56	0.49	0.49
query10	0.57	0.55	0.55
query11	0.14	0.09	0.10
query12	0.14	0.11	0.10
query13	0.63	0.62	0.63
query14	1.06	1.06	1.07
query15	0.91	0.86	0.87
query16	0.41	0.39	0.44
query17	1.11	1.15	1.14
query18	0.23	0.21	0.20
query19	2.05	2.02	2.11
query20	0.02	0.01	0.01
query21	15.39	0.22	0.15
query22	5.24	0.06	0.05
query23	16.02	0.28	0.10
query24	1.85	0.61	0.19
query25	0.09	0.05	0.09
query26	0.14	0.13	0.13
query27	0.08	0.08	0.06
query28	4.34	1.14	0.97
query29	12.59	3.97	3.21
query30	0.28	0.14	0.12
query31	2.81	0.64	0.40
query32	3.24	0.59	0.50
query33	3.34	3.24	3.33
query34	16.18	5.42	4.75
query35	4.74	4.79	4.66
query36	0.64	0.51	0.49
query37	0.11	0.07	0.07
query38	0.08	0.05	0.03
query39	0.05	0.03	0.03
query40	0.19	0.17	0.16
query41	0.09	0.03	0.03
query42	0.04	0.04	0.02
query43	0.04	0.04	0.04
Total cold run time: 98.93 s
Total hot run time: 28.25 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 27, 2026

BE UT Coverage Report

Increment line coverage 47.62% (10/21) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.49% (19160/36501)
Line Coverage 35.90% (178255/496556)
Region Coverage 32.36% (137830/425953)
Branch Coverage 33.30% (59667/179203)

@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 27, 2026

BE Regression && UT Coverage Report

Increment line coverage 47.62% (10/21) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.42% (25554/35778)
Line Coverage 54.05% (268106/496012)
Region Coverage 51.58% (222048/430458)
Branch Coverage 53.04% (95455/179971)

@freemandealer
Copy link
Contributor Author

freemandealer commented Jan 29, 2026

run performance

@freemandealer
Copy link
Contributor Author

freemandealer commented Jan 29, 2026

run p0

@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 29, 2026

BE Regression && UT Coverage Report

Increment line coverage 47.62% (10/21) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.46% (25568/35778)
Line Coverage 54.09% (268294/496012)
Region Coverage 51.60% (222120/430458)
Branch Coverage 53.09% (95542/179971)

@doris-robot
Copy link

doris-robot commented Jan 29, 2026

TPC-H: Total hot run time: 31713 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4c62152467a5aebd4a5887e5d964dbb591023a71, data reload: false

------ Round 1 ----------------------------------
q1	17658	5265	5083	5083
q2	2006	301	199	199
q3	10243	1345	744	744
q4	10269	787	316	316
q5	8743	2202	1866	1866
q6	232	183	153	153
q7	904	750	613	613
q8	9263	1422	1155	1155
q9	5327	4817	4862	4817
q10	6859	1949	1567	1567
q11	525	291	275	275
q12	406	380	237	237
q13	17801	4069	3243	3243
q14	234	244	227	227
q15	929	813	816	813
q16	675	671	620	620
q17	711	786	481	481
q18	7181	6374	6508	6374
q19	1431	1007	611	611
q20	386	339	235	235
q21	2655	2094	1808	1808
q22	363	320	276	276
Total cold run time: 104801 ms
Total hot run time: 31713 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5420	5295	5316	5295
q2	258	330	243	243
q3	2208	2689	2247	2247
q4	1389	1739	1310	1310
q5	4319	4287	4460	4287
q6	259	198	162	162
q7	2288	1916	1848	1848
q8	2631	2458	2382	2382
q9	7661	7548	7495	7495
q10	2913	3120	2654	2654
q11	554	466	456	456
q12	715	746	612	612
q13	3979	4453	3670	3670
q14	303	317	309	309
q15	907	846	842	842
q16	699	699	719	699
q17	1224	1462	1521	1462
q18	8271	7934	7646	7646
q19	881	850	851	850
q20	2082	2175	2086	2086
q21	4709	4202	4138	4138
q22	582	556	520	520
Total cold run time: 54252 ms
Total hot run time: 51213 ms

@doris-robot
Copy link

doris-robot commented Jan 29, 2026

ClickBench: Total hot run time: 28.27 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4c62152467a5aebd4a5887e5d964dbb591023a71, data reload: false

query1	0.05	0.04	0.04
query2	0.09	0.05	0.05
query3	0.25	0.08	0.08
query4	1.61	0.12	0.10
query5	0.28	0.25	0.25
query6	1.16	0.68	0.65
query7	0.03	0.03	0.03
query8	0.06	0.04	0.04
query9	0.58	0.50	0.49
query10	0.54	0.56	0.55
query11	0.14	0.09	0.10
query12	0.14	0.11	0.10
query13	0.63	0.60	0.61
query14	1.06	1.07	1.06
query15	0.88	0.86	0.87
query16	0.40	0.38	0.40
query17	1.15	1.13	1.18
query18	0.23	0.22	0.21
query19	2.12	2.03	2.01
query20	0.02	0.02	0.01
query21	15.40	0.27	0.16
query22	5.15	0.06	0.05
query23	15.93	0.29	0.10
query24	2.39	0.59	0.22
query25	0.10	0.07	0.07
query26	0.14	0.13	0.14
query27	0.07	0.08	0.06
query28	5.05	1.14	0.97
query29	12.54	3.98	3.19
query30	0.28	0.13	0.12
query31	2.82	0.63	0.41
query32	3.25	0.59	0.50
query33	3.26	3.23	3.27
query34	16.42	5.34	4.72
query35	4.80	4.76	4.82
query36	0.66	0.50	0.48
query37	0.10	0.07	0.07
query38	0.07	0.04	0.04
query39	0.04	0.02	0.03
query40	0.18	0.15	0.15
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.05	0.04	0.03
Total cold run time: 100.24 s
Total hot run time: 28.27 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou Awaiting requested review from gavinchou gavinchou is a code owner

@dataroaring dataroaring Awaiting requested review from dataroaring dataroaring is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@freemandealer @Thearas @doris-robot @hello-stephen