feat(retain): make semantic conversation chunking the default

[Danielxu0208]

Motivation

Closes #5

PR #6 has been merged into main. This branch is rebased onto that merged implementation so the diff now contains only semantic conversation chunking and its integration hardening.

Long JSON conversations previously used size-based boundaries that could separate a user turn from its response and ignored topic transitions. This change makes semantic boundary planning the default while preserving source integrity, atomic publication, and deterministic recovery.

Changes

• Parse strict role/content JSON conversations into complete user exchanges.
• Ask the Retain model for ordered end-exchange indices only; the model never supplies rewritten chunk text.
• Materialize every chunk deterministically from the original turns, preserving order, fields, and values.
• Keep oversized individual exchanges atomic and mark them explicitly.
• Bypass semantic planning for short content, trusted pre-chunked input, provider-none paths, and non-conversation content.
• Support configurable fixed_fallback and fail-closed raise policies, with bounded retries and completion tokens.
• Persist versioned, text-free semantic manifests with source hashes, chunk hashes/positions, policy fingerprints, and plan digests.
• Reuse compatible durable plans without another provider call; reject tampered or incompatible recovery state.
• Force a conservative FULL plan when semantic layout or policy changes, while preserving DELTA behavior for compatible unchanged chunks.
• Run provider planning outside database snapshots with sibling cancellation and token accounting.
• Preserve semantic boundaries through fact extraction and keep multi-window FULL publication atomic with the merged PR refactor(retain): replace orchestration with recoverable ingestion #6 transaction/checkpoint/outbox rules.
• Record truthful LongMemEval chunking provenance: fresh banks record the current policy, reused banks mark the creator policy unverifiable, and mixed runs separate current-run policy from reused-bank provenance.
• Add semantic planner, service, extraction-boundary, recovery, manifest, and semantic-plus-atomic rollback tests to the offline quality gate.

Validation

• uv lock --check: passed.
• Locked dependency sync: passed.
• Core Ruff check and format check: passed.
• LongMemEval Ruff check and format check: passed.
• Compile/import smoke and benchmark launcher syntax: passed.
• Retain offline contract suite: 309 passed, 3 skipped (live-only local skips).
• LongMemEval offline suite: 80 passed.
• Semantic + atomic rollback regression: 2 passed (second-window cancellation and write failure).
• Wheel and sdist build: passed; both contain LICENSE and THIRD_PARTY_NOTICES.md.
• Secret, local-path, generated-artifact, conflict-marker, and diff-scope audit: passed.
• GitHub Actions: Python 3.11 offline, PostgreSQL 16, and Oracle 23ai all passed.

Risk / Rollback

• Semantic planning adds one provider call for eligible long conversations, increasing latency and cost.
• Invalid provider output follows the configured fixed_fallback or raise policy.
• Provider Batch extraction remains incompatible with an uncommitted semantic plan because its checkpoint does not yet bind the plan digest.
• Enabling semantic chunking by default changes chunk identities and may require a conservative FULL refresh.
• A single oversized exchange may exceed the configured chunk size so that the exchange remains intact.
• Runtime rollback: set HMS_API_RETAIN_SEMANTIC_CHUNKING_ENABLED=false to restore deterministic chunking.
• Code rollback: disable the feature, then revert this PR. Existing semantic documents may receive a conservative FULL refresh on their next update.