perf: optimize full-text index construction and document AI functions

[Ludwig-LMY]

Summary

• optimize full-text index construction hot paths, including DDL scan expression reuse, persistent LOB allocation, and full requested auxiliary-task parallelism
• cache deterministic TOKENIZE results with dictionary-epoch invalidation
• use predicate-only MATCH execution paths that avoid unnecessary BM25 statistics/relevance work while preserving projection, ordering, comparison, and BOOLEAN MODE semantics
• implement and harden LOAD_FILE and AI_SPLIT_DOCUMENT, including public function registration, correct BLOB/LOB result construction, and function-table string collation metadata

Official FTS benchmark

Ran the checked-in tools/benchmark/fts_large_bench.sh with the fixed baseline configuration:

• score: 62.57 / 100
• mean improvement: 31.29%
• build improvement: 23.21%
• tokenize improvement: 19.64%
• query improvement: 51.00%
• expected hit counts preserved: 8001 / 11000 / 7332 / 20

Verification

• Debug observer target builds and links successfully
• LOAD_FILE returns the expected file content and byte length
• AI_SPLIT_DOCUMENT sentence, sliding-window word, and Markdown cases match expected output
• MATCH predicate, relevance projection/order, numeric score comparison, and BOOLEAN MODE smoke tests pass
• no benchmark tests or GitHub workflow files were modified

[CLAassistant]

All committers have signed the CLA.

[LINxiansheng]

Document AI & IK Custom Dictionary Score

Document AI Functions Score
===========================
score: 100.00 / 100
load_file: 50 / 50
ai_split_document: 50 / 50

IK Custom Dictionary Score
==========================
score: 0.00 / 100
ik_custom_dict: 0 / 100

FTS Large Benchmark Score

FTS Large Benchmark Score
=========================
score: 51.52 / 100
mean_improvement: 25.76%
full_score_improvement: 50.00%

build_improvement: 0.40%
  build_ik_all_sec: baseline=35.2836, current=35.315, improvement=-0.09%
  build_ik_content_sec: baseline=28.3764, current=28.149, improvement=0.80%
  build_beng_en_sec: baseline=14.7578, current=14.686, improvement=0.49%
tokenize_improvement: 37.12%
  tokenize_ik_avg_ms: baseline=0.76478, current=0.3637, improvement=52.44%
  tokenize_beng_avg_ms: baseline=0.42262, current=0.3305, improvement=21.80%
query_improvement: 39.77%
  query_cn_avg_ms: baseline=16.6628, current=7.5659, improvement=54.59%
  query_beng_avg_ms: baseline=24.3042, current=11.4043, improvement=53.08%
  query_mixed_avg_ms: baseline=17.5593, current=8.5872, improvement=51.10%
  query_limit_avg_ms: baseline=16.2334, current=16.185, improvement=0.30%

FTS Large Benchmark Report

========================================
FTS Large Benchmark Report
========================================
timestamp:              2026-07-18 08:25:08
label:                  vldb-ci-29636561999-1
git_head:               0c52f08
git_dirty:              0
rows:                   20000
batch:                  500
rounds:                 3000
query_rounds:           200
samples:                3
warmup:                 30
skip_load:              0
----------------------------------------
select1_avg_ms:         0.2673
select1_stdev_ms:       0.0035
raw_load_sec:           1.580
raw_load_rows_per_sec:  12658.2
build_ik_all_sec:       35.315
build_ik_content_sec:   28.149
build_beng_en_sec:      14.686
build_total_sec:        78.163
----------------------------------------
tokenize_ik_avg_ms:     0.3637
tokenize_ik_median_ms:  0.3642
tokenize_ik_stdev_ms:   0.0016
tokenize_beng_avg_ms:   0.3305
tokenize_beng_median_ms:0.3313
tokenize_beng_stdev_ms: 0.0011
----------------------------------------
query_cn_hits:          8001
query_cn_avg_ms:        7.5659
query_cn_stdev_ms:      0.0018
query_beng_hits:        11000
query_beng_avg_ms:      11.4043
query_beng_stdev_ms:    0.0058
query_mixed_hits:       7332
query_mixed_avg_ms:     8.5872
query_mixed_stdev_ms:   0.0008
query_limit_hits:       20
query_limit_avg_ms:     16.1850
query_limit_stdev_ms:   0.0193
========================================

Workflow run