fix(store): index BM25 on a search_text column carrying the abstract and summary layers

[gorkem2020]

Summary

Fixes #913. The BM25 index sat on the text column, which only holds the L0 abstract, so a keyword that survived only in the overview or the content layer was invisible to indexed full-text search. The lexical scan did see those layers, but only when the indexed query returned nothing at all. The vector column already embeds the abstract plus the content and is untouched.

Changes

• src/search-text.ts (new): buildSearchText(text, metadata) joins the abstract with the l0_abstract, l1_overview and l2_content metadata layers, folding identical layers once; missing or broken metadata degrades to the abstract.
• src/store.ts: new search_text column and the FTS index now targets it. Every write path (store/bulkStore, upsert, importEntry, update, bulkUpdateExact, the recall-metadata merge, the legacy-scope repair, storeSuperseding) passes rows through one toRow boundary that recomputes the column from the row's own text and metadata, so no rewrite path can let it drift. Tables from before the column existed get it through the existing legacy-column migration: addColumns seeds it with the abstract, then the same locked step backfills it from the metadata layers. createFtsIndex moves the index off the abstract column and drops the leftover index so a bare FTS query stays unambiguous; bm25Search names the column explicitly; rebuildFtsIndex recreates on the current column. A store whose migration did not run keeps the previous text index (the column is resolved from the schema and the index list, never assumed).
• MemoryStore.backfillSearchText({ dryRun }): report or rewrite rows whose column is missing or stale. It checks out the latest table version first, so a CLI process sees the gateway's writes.
• cli.ts: memory-pro backfill-search-text (report only; --apply rewrites), the repair path next to reindex-fts.
• Tests: test/fts-search-text-column.test.mjs covers the builder, a content-only token found through the real index with the lexical fallback forbidden, the rewrite paths keeping the column current, a pre-upgrade table being migrated, backfilled and re-indexed on open, the backfill reporting and repairing a stale row, and reindex-fts ending with a single index on search_text. test/fts-index-fold.test.mjs and the CLI attachment test follow the column move. Registered in the npm test chain and the CI manifest.

Notes

• LanceDB 0.26 evaluates addColumns value expressions against existing columns only, so the schema step can seed the abstract but cannot parse the metadata JSON; the app-level backfill in the same migration step does that.
• On 0.26 a mergeInsert(...).whenMatchedUpdateAll() source row that lacks the column keeps the target value, while add without it writes NULL; the write boundary sets the column on every path, so the behavior does not depend on either.
• The result shape is unchanged: entry.text keeps returning the abstract; the new column is index plumbing only.

Verification

• npm run build, npm test (full chain), the new test file, fts-index-fold, store-lexical-metadata-search, cli-subcommand-attachment, redis-lock, migrate-legacy-schema, node scripts/verify-ci-test-manifest.mjs.

[gorkem2020]

Ready for review. Rebased onto the current master (93899f88, post #946); only the two test-registration files conflicted (ordered union), the source is unchanged from the draft.

Fleet evidence: this change has run on our gateway since 2026-09-12 (the store migration ran once on first start and backfilled search_text for every existing row, index search_text_idx), with hybrid and FTS retrieval healthy through today. npm run build and the full npm test chain pass on the rebased head.