feat: implement Document AI functions & support custom IK dictionaries#1064
Open
breeze-wink wants to merge 17 commits into
Open
feat: implement Document AI functions & support custom IK dictionaries#1064breeze-wink wants to merge 17 commits into
breeze-wink wants to merge 17 commits into
Conversation
Add LOAD_FILE support for reading BLOB data from local locations and implement AI_SPLIT_DOCUMENT as a table function with text, markdown, sentence, word, max, and overlap handling. Register the parser, resolver, expression, and execution paths, and add a one-command runner for the official mysqltests. Tests: ./tools/deploy/run_document_ai_tests.sh --no-build
Member
Document AI & IK Custom Dictionary ScoreDocument AI Functions Score =========================== score: 100.00 / 100 load_file: 50 / 50 ai_split_document: 50 / 50 IK Custom Dictionary Score ========================== score: 100.00 / 100 ik_custom_dict: 100 / 100 FTS Large Benchmark ScoreFTS Large Benchmark Score ========================= score: 100.00 / 100 mean_improvement: 55.07% full_score_improvement: 50.00% build_improvement: 44.51% build_ik_all_sec: baseline=35.2836, current=19.499, improvement=44.74% build_ik_content_sec: baseline=28.3764, current=15.281, improvement=46.15% build_beng_en_sec: baseline=14.7578, current=8.466, improvement=42.63% tokenize_improvement: 46.55% tokenize_ik_avg_ms: baseline=0.76478, current=0.3415, improvement=55.35% tokenize_beng_avg_ms: baseline=0.42262, current=0.2631, improvement=37.75% query_improvement: 74.15% query_cn_avg_ms: baseline=16.6628, current=5.3184, improvement=68.08% query_beng_avg_ms: baseline=24.3042, current=7.3896, improvement=69.60% query_mixed_avg_ms: baseline=17.5593, current=5.7707, improvement=67.14% query_limit_avg_ms: baseline=16.2334, current=1.3352, improvement=91.77% FTS Large Benchmark Report======================================== FTS Large Benchmark Report ======================================== timestamp: 2026-07-17 16:28:29 label: vldb-ci-29593716819-1 git_head: e690d7a git_dirty: 0 rows: 20000 batch: 500 rounds: 3000 query_rounds: 200 samples: 3 warmup: 30 skip_load: 0 ---------------------------------------- select1_avg_ms: 0.2043 select1_stdev_ms: 0.0037 raw_load_sec: 1.464 raw_load_rows_per_sec: 13661.2 build_ik_all_sec: 19.499 build_ik_content_sec: 15.281 build_beng_en_sec: 8.466 build_total_sec: 43.263 ---------------------------------------- tokenize_ik_avg_ms: 0.3415 tokenize_ik_median_ms: 0.3617 tokenize_ik_stdev_ms: 0.0293 tokenize_beng_avg_ms: 0.2631 tokenize_beng_median_ms:0.2639 tokenize_beng_stdev_ms: 0.0022 ---------------------------------------- query_cn_hits: 8001 query_cn_avg_ms: 5.3184 query_cn_stdev_ms: 0.0010 query_beng_hits: 11000 query_beng_avg_ms: 7.3896 query_beng_stdev_ms: 0.0041 query_mixed_hits: 7332 query_mixed_avg_ms: 5.7707 query_mixed_stdev_ms: 0.0031 query_limit_hits: 20 query_limit_avg_ms: 1.3352 query_limit_stdev_ms: 0.0017 ======================================== |
breeze-wink
changed the title
Author
|
Hunan-Guangxi Fans Team |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Task Description
2026数据库暑期学校测试：湘桂粉丝队
Release Note