feat: implement Document AI functions and IK custom dictionary support

[jbjvhvhh]

Task Description

Implement and integrate Document AI Functions and IK custom dictionary support for the VLDB 2026 tasks.

Solution Description

• Implement LOAD_FILE.
• Implement AI_SPLIT_DOCUMENT.
• Support IK custom dictionaries, stopword dictionaries and quantifier dictionaries.
• Resolve parser item type conflicts between the two implementations.
• Preserve the full-text tokenizer performance optimizations.

Test

• Full observer build passed.
• load_file: passed.
• ai_split_document: passed.
• ik_custom_dict: passed.
• Full-text benchmark query results are correct.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
0 out of 2 committers have signed the CLA.

❌ jbjvhvhh
❌ bluejunhao

---

jbjvhvhh seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[LINxiansheng]

Document AI & IK Custom Dictionary Score

Document AI Functions Score
===========================
score: 100.00 / 100
load_file: 50 / 50
ai_split_document: 50 / 50

IK Custom Dictionary Score
==========================
score: 100.00 / 100
ik_custom_dict: 100 / 100

FTS Large Benchmark Score

FTS Large Benchmark Score
=========================
score: 91.75 / 100
mean_improvement: 45.88%
full_score_improvement: 50.00%

build_improvement: 43.02%
  build_ik_all_sec: baseline=35.2836, current=21.281, improvement=39.69%
  build_ik_content_sec: baseline=28.3764, current=17.249, improvement=39.21%
  build_beng_en_sec: baseline=14.7578, current=7.355, improvement=50.16%
tokenize_improvement: 35.49%
  tokenize_ik_avg_ms: baseline=0.76478, current=0.3645, improvement=52.34%
  tokenize_beng_avg_ms: baseline=0.42262, current=0.3438, improvement=18.65%
query_improvement: 59.12%
  query_cn_avg_ms: baseline=16.6628, current=8.0745, improvement=51.54%
  query_beng_avg_ms: baseline=24.3042, current=12.0292, improvement=50.51%
  query_mixed_avg_ms: baseline=17.5593, current=9.0588, improvement=48.41%
  query_limit_avg_ms: baseline=16.2334, current=2.2709, improvement=86.01%

FTS Large Benchmark Report

========================================
FTS Large Benchmark Report
========================================
timestamp:              2026-07-18 06:01:37
label:                  vldb-ci-29632681800-1
git_head:               4d9fed8
git_dirty:              0
rows:                   20000
batch:                  500
rounds:                 3000
query_rounds:           200
samples:                3
warmup:                 30
skip_load:              0
----------------------------------------
select1_avg_ms:         0.2672
select1_stdev_ms:       0.0028
raw_load_sec:           1.495
raw_load_rows_per_sec:  13377.9
build_ik_all_sec:       21.281
build_ik_content_sec:   17.249
build_beng_en_sec:      7.355
build_total_sec:        45.899
----------------------------------------
tokenize_ik_avg_ms:     0.3645
tokenize_ik_median_ms:  0.3641
tokenize_ik_stdev_ms:   0.0031
tokenize_beng_avg_ms:   0.3438
tokenize_beng_median_ms:0.3334
tokenize_beng_stdev_ms: 0.0210
----------------------------------------
query_cn_hits:          8001
query_cn_avg_ms:        8.0745
query_cn_stdev_ms:      0.0136
query_beng_hits:        11000
query_beng_avg_ms:      12.0292
query_beng_stdev_ms:    0.0094
query_mixed_hits:       7332
query_mixed_avg_ms:     9.0588
query_mixed_stdev_ms:   0.0052
query_limit_hits:       20
query_limit_avg_ms:     2.2709
query_limit_stdev_ms:   0.0042
========================================

Workflow run