feat: index webpages

[akiva10b]

Summary

flowchart LR
    A["Sefaria Linker"] --> B["MongoDB"]
    B --> C["Celery indexing task"]
    C --> D["Clean and chunk content"]
    D --> E["Generate embeddings"]
    E --> F["Elasticsearch webpage index"]
    G["AI agent search tool"] --> H["Search API"]
    H --> F

Loading

This PR adds an agent-facing search service for webpage content collected by the Sefaria Linker. It supports:

• Corpus-wide and website-scoped searches
• Filtering by Sefaria ref, domain, website, and language
• Lexical, semantic, and hybrid retrieval
• Passage-level results suitable for AI agent consumption

Architecture

MongoDB remains the source of truth:

• webpages stores page metadata and detected refs.
• webpages_text stores the extracted title and full page content.

When WebPageText changes, a Celery task cleans and chunks the page content, generates embeddings, and indexes each passage in a dedicated Elasticsearch index. Existing chunks for the URL are replaced when its content changes.

Each indexed passage includes:

• Page and chunk identifiers
• URL, domain, and website ID
• Title and passage content
• Original and expanded Sefaria refs
• Detected language
• Last-updated timestamp
• Vector embedding

Website IDs, domains, refs, expanded refs, and languages are indexed as exact-match fields. Titles and content are indexed for full-text retrieval.

Search behavior

The search service supports:

• BM25 lexical search for quotations, names, and precise terminology
• Vector search for conceptual matching
• Hybrid lexical and semantic retrieval
• Exact filters for website, domain, language, and Sefaria ref
• URL-level result grouping to prevent a single page from dominating results

Ref filters are normalized through Ref. Sections and ranges are expanded to segment refs and matched against expanded_refs.

Page-level refs are initially copied to every passage from that page. A future improvement could associate refs only with the passages where they occur.

Agent API

The bounded search interface accepts:

search_webpages(
    query: str,
    website_id: str | None = None,
    domain: str | None = None,
    ref: str | None = None,
    language: str | None = None,
    mode: Literal["hybrid", "lexical", "semantic"] = "hybrid",
    limit: int = 10,
)

Results include the page title, URL, matching passage, relevance score, website metadata, and associated refs. Domain and ref normalization are handled by the API rather than the calling agent.

Operations

The implementation uses Sefaria’s existing Elasticsearch, MongoDB, Celery, and embedding infrastructure. It includes:

• A separate webpage search index
• Bulk and incremental indexing
• Configurable indexing on linker updates
• A full reindex command
• MongoDB-backed index reconstruction
• Tests for chunking, filtering, ref expansion, hybrid retrieval, and result formatting

Hybrid retrieval is the default, website_id is the canonical website filter, domain provides hostname-level filtering, and expanded_refs supports segment-level ref filtering.

[gitvelocity-reviewer]

📊 Code Quality Score: 50/100

63 × 0.8 = 50.4 → 50

Category	Score	Factors
🔭 Scope	14/20	New module + API endpoint + Celery task + reindex script touching linker, views, URLs, tasks, settings; new public API endpoint (+2); external ES integration (+2); cross-cutting within one feature domain
🏗️ Architecture	13/20	New helper module with clean separation of indexing/retrieval; introduces hybrid search pattern as new architectural component; follows existing patterns from sefaria/helper/search.py; Celery task integration follows existing patterns
⚙️ Implementation	14/20	Hybrid search with lexical+vector modes; chunking with overlap and deterministic SHA-256 IDs; HTML stripping via bleach; Elasticsearch mapping with dense_vector; filter building with ref expansion; reasonable error handling
⚠️ Risk	10/20	New ES index is additive; feature-flagged with WEBPAGE_SEARCH_INDEX_ON_SAVE=False default (-2); WebPageText.add_or_update contract change is undocumented risk (+3); bearer token auth for new endpoint (+2); no rollback plan documented
✅ Quality	9/15	Good test coverage of core logic (~65%, below 70% threshold); missing tests for index_webpage_text, create_webpage_search_index, API view auth/error paths, Celery task; settings documented in example and CI files; no API documentation
🔒 Perf / Security	3/5	Bearer token auth implemented; double ES refresh (delete+bulk) could be expensive at scale; no rate limiting on search endpoint; input validation present for query/mode/limit

Was this score accurate? 👍 Yes · 👎 No

How this was scored →

_{Scored by GitVelocity · How are scores calculated?}