refactor(retain): replace orchestration with recoverable ingestion

[Danielxu0208]

Motivation

Closes #4

Retain needs explicit transaction and recovery boundaries, consistent PostgreSQL and Oracle behavior, and reproducible quality evidence. This PR replaces the monolithic orchestration path with a database-neutral ingestion service and adds the tests, benchmark tooling, and CI gates required to review that change.

Semantic conversation chunking is intentionally excluded. It follows in #5 and stacked PR #7.

Changes

• Replace retain/orchestrator.py with layered planning, extraction, projection, persistence, and execution components.
• Plan documents from one read-only snapshot, then apply hash-guarded full, delta, metadata-only, or stale-skip writes.
• Add bounded write windows, durable checkpoints, operation fences, cancellation handling, and retry-stable logical document grouping.
• Publish all FULL write windows in one transaction, validate their complete mapping before checkpoint/outbox publication, and roll back the entire replacement on cancellation, ownership loss, validation failure, or downstream write failure.
• Roll back Oracle transactions on asyncio.CancelledError as well as ordinary exceptions.
• Preserve original input order and one-to-one Retain result mapping across token batches.
• Add PostgreSQL and Oracle persistence adapters plus focused contract and live regression tests.
• Add the HMS_API_RETAIN_EMBEDDING_FAILURE_POLICY setting with store_without_embedding and raise modes.
• Fix Oracle projection backfill execution by using raw driver SQL and DBMS_LOB.COMPARE for CLOB values.
• Add a pinned and resumable LongMemEval Retain -> Recall -> Answer -> Judge workflow with integrity checks, bounded concurrency, source audits, and reproduction documentation.
• Require an exact document/content/metadata audit before reusing a LongMemEval bank; record reused or mixed bank creator provenance as unverifiable instead of attributing it to the current runtime.
• Enable Recall trace collection for cached/reranked diagnostics while excluding trace data from answer-generation inputs.
• Add offline, PostgreSQL, and Oracle quality gates with pinned actions and service images.
• Package the required license and third-party notices.

Validation

• uv lock --check : passed, 267 packages resolved from the lockfile.
• Retain Ruff check : passed.
• Retain Ruff format check : passed, 57 files already formatted.
• LongMemEval Ruff check : passed.
• LongMemEval Ruff format check : passed, 12 files already formatted.
• Compile and import smoke test : passed.
• bash -n .aaaSCRIPT/run_benchmark.sh : passed.
• Retain offline contract suite : 255 passed, 3 skipped.
• LongMemEval offline suite : 71 passed.
• Wheel and source distribution build : passed.
• Wheel and source distribution license notice check : passed.
• GitHub Actions offline quality gate : passed.
• PostgreSQL 16 live smoke : passed in GitHub Actions.
• Oracle 23ai live smoke : passed in GitHub Actions.

A paid 500-question LongMemEval run is not required for this PR. This change makes that run reproducible but does not claim a benchmark score.

Risk / Rollback

• Risk: This PR changes internal Retain transaction, retry, cancellation, checkpoint, and result-mapping behavior. A persistence adapter regression can affect document, fact, graph, or operation state.
• Risk: Prebuilding every FULL write window before the atomic transaction increases peak memory usage compared with committing windows independently.
• Risk: The default embedding failure policy permits facts without vectors. Deployments that require fail-closed ingestion must set HMS_API_RETAIN_EMBEDDING_FAILURE_POLICY=raise.
• Risk: The Oracle fix changes an existing Alembic revision. An already-applied revision will not rerun automatically.
• Risk: LongMemEval artifact schema v2 is intentionally not resume-compatible with older artifacts; use a new HMS_RESULTS_FILENAME for retrieval-only output.
• Risk: LongMemEval sends configured inputs to external model endpoints and can incur substantial cost.
• Rollback: Revert the squash commit and redeploy the previous Retain path. Inspect operations left in pending or processing before retrying asynchronous work. Review Oracle migration state separately before any manual backfill.

[kawhiiiileo]

Thanks for the substantial refactor. I found two blocking correctness issues that need to be addressed before this can be merged:

1. A tracked FULL ingestion can leave a partial replacement visible after cancellation between write windows. _execute_full_document_windows() commits each window independently. The first window calls handle_document_tracking(), which deletes the previous document and its memory units, writes only the first window, and changes the document hash to retain-inflight:*. If cancellation is accepted before the next window, OperationActivityFence rejects that next write as terminal cancellation, with no retry or cleanup. Recall does not filter out in-flight documents, so the partial replacement remains queryable and the previously valid version has already been lost.

   Please stage the windows and publish atomically, preserve the old version until final publication, or add transactional compensation/cleanup for accepted cancellation. Please also add a live or contract regression test using an existing multi-chunk document and retain_chunk_batch_size=1, cancelling after the first window commits and asserting that no partial replacement is visible and the previous version is not destroyed.

2. LongMemEval retrieval-only reuse does not establish the provenance it records in the artifact. _agent_has_data() and _audit_durable_ingestion() verify only that chunks/documents with the expected IDs exist; they do not compare documents.content_hash with the selected dataset content or validate the Retain pipeline/model/code identity that created the bank. A stale bank from another dataset revision or Retain configuration can therefore pass reuse checks, while the output manifest attributes the result to the current dataset SHA and model configuration.

   Please validate each expected document content hash before reuse and persist/validate the relevant ingestion pipeline, model, and code identity (or explicitly mark reused-bank provenance as unverifiable).

One additional non-blocking issue: recall_async() is called without enable_trace=True, so search_result.trace is always empty and the cached coarse/reranked diagnostic candidates are never populated.

Please continue updating the PR and add regression coverage for these cases. I’ll re-review once the fixes are pushed.

[Danielxu0208]

@kawhiiiileo Thanks for the review. I addressed all three items:

1. Atomic FULL publication: all FULL write windows now publish in one transaction. Cancellation, ownership loss, invalid result mapping, checkpoint failure, and outbox failure roll back the whole replacement, preserving the previous document version. Oracle transaction scopes now also roll back on asyncio.CancelledError. Added the requested existing-document, multi-chunk, retain_chunk_batch_size=1 regression.
2. LongMemEval reuse provenance: reused banks now require an exact document/content-hash/metadata/chunk-state audit before QA. Reused or mixed banks explicitly report their creator Retain/model/code identity as unverifiable instead of attributing them to the current runtime. Artifact schema is now v2.
3. Recall diagnostics: recall_async now runs with enable_trace=True; cached and reranked diagnostics are populated from the trace, and the trace itself is excluded from answer-generation input.

Validation:

• Retain offline suite: 255 passed, 3 skipped
• LongMemEval offline suite: 71 passed
• Ruff, format, compile/import, lockfile, packaging, and notice checks: passed
• GitHub Actions offline, PostgreSQL 16, and Oracle 23ai checks: all passed

Fix commits: 011a0f2 and 675f4d3.

Could you please re-review PR #6?