implement LOAD_FILE，AI_SPLIT_DOCUMENT and OceanBase full-text index building performance optimization#1062
Open
Huiqin-Zhang wants to merge 21 commits into
Open
Conversation
…uilding performance optimization Team：南北小队
|
|
Register AI_SPLIT_DOCUMENT as a non-reserved keyword and use a dedicated grammar token so Bison generation succeeds without NAME_OB table-function conflicts.
Restore REFRESH MEMORY STAT parsing and remove the invalid $6 reference from its five-symbol production.
Restore the allocator-backed list for IK processors and add string serialization for split units used by diagnostic array formatting.
Provide valid local storage metadata to LOAD_FILE and own IK dictionary table names instead of retaining strings from a temporary JSON document.
Member
Document AI & IK Custom Dictionary ScoreDocument AI Functions Score =========================== score: 100.00 / 100 load_file: 50 / 50 ai_split_document: 50 / 50 IK Custom Dictionary Score ========================== score: 100.00 / 100 ik_custom_dict: 100 / 100 FTS Large Benchmark ScoreFTS Large Benchmark Score ========================= score: 14.39 / 100 mean_improvement: 7.20% full_score_improvement: 50.00% build_improvement: 12.15% build_ik_all_sec: baseline=35.2836, current=30.564, improvement=13.38% build_ik_content_sec: baseline=28.3764, current=24.678, improvement=13.03% build_beng_en_sec: baseline=14.7578, current=13.276, improvement=10.04% tokenize_improvement: 9.15% tokenize_ik_avg_ms: baseline=0.76478, current=0.7135, improvement=6.71% tokenize_beng_avg_ms: baseline=0.42262, current=0.3736, improvement=11.60% query_improvement: 0.29% query_cn_avg_ms: baseline=16.6628, current=16.6455, improvement=0.10% query_beng_avg_ms: baseline=24.3042, current=24.2548, improvement=0.20% query_mixed_avg_ms: baseline=17.5593, current=17.3963, improvement=0.93% query_limit_avg_ms: baseline=16.2334, current=16.2458, improvement=-0.08% FTS Large Benchmark Report======================================== FTS Large Benchmark Report ======================================== timestamp: 2026-07-16 09:02:37 label: vldb-ci-29484160581-1 git_head: 0cd149e git_dirty: 0 rows: 20000 batch: 500 rounds: 3000 query_rounds: 200 samples: 3 warmup: 30 skip_load: 0 ---------------------------------------- select1_avg_ms: 0.2135 select1_stdev_ms: 0.0041 raw_load_sec: 1.500 raw_load_rows_per_sec: 13333.3 build_ik_all_sec: 30.564 build_ik_content_sec: 24.678 build_beng_en_sec: 13.276 build_total_sec: 68.531 ---------------------------------------- tokenize_ik_avg_ms: 0.7135 tokenize_ik_median_ms: 0.7066 tokenize_ik_stdev_ms: 0.0173 tokenize_beng_avg_ms: 0.3736 tokenize_beng_median_ms:0.3736 tokenize_beng_stdev_ms: 0.0004 ---------------------------------------- query_cn_hits: 8001 query_cn_avg_ms: 16.6455 query_cn_stdev_ms: 0.0166 query_beng_hits: 11000 query_beng_avg_ms: 24.2548 query_beng_stdev_ms: 0.0262 query_mixed_hits: 7332 query_mixed_avg_ms: 17.3963 query_mixed_stdev_ms: 0.0064 query_limit_hits: 20 query_limit_avg_ms: 16.2458 query_limit_stdev_ms: 0.0117 ======================================== |
Port the optimized parser, dictionary, token processing, and parser-state reuse paths. Preserve vldb_2026 include layout and custom IK dictionary support. Team: 南北小队
Use the branch-local datum hash type and include the complete parser descriptor definition.
Stop dividing the sequential auxiliary build parallelism and keep IK processor pointers in inline storage.
Keep the optimized FTS implementation compatible with the vldb_2026 source layout. Team: 南北小队
Use the vldb_2026 ObSEArray reset API and provide printable processor metadata. Team: 南北小队
Parse custom main, quantifier, and stopword table names into the reusable parser context. Team: 南北小队
Huiqin-Zhang
force-pushed
the
feat/vldb-2026-solutions
branch
from
July 15, 2026 03:32
8b96150 to
308c162
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
提交测试