feat(memory): add compact Atomic Memory directory

[wutongyuonce]

Which issue or RFC does this PR close?

Refs #1656. Follows the Atomic Memory/shared Artifact model in #1809/#1859 and the explicit migration boundary in #1771.

Rationale for this change

Browsing Atomic Memory identities should not load complete content or history. This adds a compact current-head directory with exact Artifact references. A content, lifecycle, merge/restoration or effective tag change invalidates continuation; callers restart from page one.

What changes are included in this PR?

• Add POST /v1/atomic-memory/directory/query, query_atomic_memory_directory in the HTTP SDK/MCP, and the Python scoped query. OpenAPI owns the generated contracts; Codex session Scope binding includes the operation. Regenerate Desktop wire/API contracts and the distributed DSH bundle; include the additive tool in Skill mock catalogs and negative controls.
• Read compact shared heads with state/kind/tag selection before keyset limits, signed query/subject/policy/generation cursors, and response budgets. Directory requests expand no body, Source or history. Formal owner readiness still covers the whole Scope; SQLite may scan it, so this is not a constant database-work claim.
• Add no pagination tables. SQLite writer locks/increments reject noninteger stored generation values, preventing publication with a damaged counter. Extend Scope generation and head kind through frozen resources, and maintain both through shared publication, governance, tags and merge/restoration transactions. Retain caller-owned transactions and defer audit writes. All directory-enabled lock paths acquire the Scope boundary before heads/content.
• Register the optional persistent SQLite directory target in the existing server db-migrate entry point and single revision table. Append frozen p0005/p0006 resources on top of the released p0004, current-head backfill/validation, shared batch commits and original-backup recovery. The default five-table acceptance target stays at the released p0004; released revision/env bytes are unchanged.
• Keep ordinary startup from adding or backfilling directory fields. Unfinished recorded maintenance prevents initializer writes. A structurally valid managed directory retains writer maintenance while damaged data disables querying. Topic-only child runtimes do not scan unrelated Atomic bodies.
• Update bilingual workflow documentation, the optional MCP tool declaration, and migration/application/transport regressions.

Are there any user-facing changes?

The additive directory returns compact exact references with default limit 50 / maximum 100. Continuation must restart after changes. Full-record APIs retain their current contracts.

Production enablement in this PR is restricted to the registered default current SQLite business layout, ordinary persistent files, no embedding extension, and trusted-local or qualified same-database Builtin/Casbin Scope-read authorization. Stop all writers, run server db-migrate plan/apply/verify --target atomic-memory-directory --env-file deployment.env, then restart the matching program. The target uses the database-adjacent default evidence directory. Migration does not invent owners.

Unmanaged targets and unsupported access configurations fail explicitly. Unregistered historical/archive layouts, vector configurations and seekDB/OceanBase directory execution are not supported by this target. Existing initializers in other domains remain unchanged; this PR does not complete the entire migration RFC rollout. Shared migration output remains readiness_scope=registered_bundle, server_ready=false. Restore under stopped writes and rotate the existing cursor signing secret before resuming.

How was this change tested?

• Merge-head .venv/bin/python -m pytest -n 2 --doctest-modules tests -q: 5,380 passed, 429 skipped. Skips/deselections do not establish external backend, native-host or real-model acceptance.
• Final-head directory storage/application/migration core on Python 3.11, 3.12 and 3.14: 45 passed on each version, including the integer-storage guard regression. Current directory/migration/Handoff/API focused tests: 137 passed. Python 3.13 is covered by the full run; full four-version tox was not run.
• Formal API generation and Python/JS contract tests: 52 passed. All 11 pre-commit hooks and full ty check passed.
• Final-head hosted DSH package, native personal service and Windows Desktop installer/smoke checks passed. Skill guidance: 28 local tests, checksum-verified skill-up v0.12.0 schema validation and dry-run passed.
• make docs-test: lint/test/static build passed, with 921 pages and internal links verified. sdist/wheel builds passed; packaged code and frozen resources were checked against the source.
• Installed final wheel in an isolated environment and ran the unified CLI against a fresh SQLite target. A real listening Server with enforced authentication passed HTTP SDK first-page, network MCP continuation/read-only tool metadata, exact revision reads, write-invalidated cursor and missing-token 401 checks.
• Actual Server-created old business databases preserve exact history and Server identity across migration; 201 records exercise multiple committed batches. Interrupted and bad later batches retain the original recovery point, resume safely, and do not manufacture changes on repeated success. Post-completion data damage fails validation while managed writers retain generation maintenance.
• Eight concurrent same-Scope writes produced eight records and generation 8. The keyset plan uses the existing composite primary-key index. A native SQLite snapshot/restore plus signing-key rotation rejects pre-restoration cursors.

Final-head GitHub CI: 35 checks passed, 0 failed, 0 pending, including Python 3.11–3.14 unit/e2e, hosted quality, Skill guidance, ZCode, Windows Desktop, DSH, native services, documentation and existing backend acceptance/clean-migration checks. CLA status also passed. The separate optional Desktop installed-package job was skipped and is not counted as passed; the Windows job built the installer and ran its installed-package smoke.

Validation limit: make check (release/manifest/guidance checks, uv lock --locked, all prek hooks, ty check and the pydantic-ai ty check) passes on the merge head when uv resolves the repository's own index. The earlier local failure came from a local uv default index, which rewrote every lock URL; a mirror-based run is not a dependency change, and this contribution neither adds nor updates any dependency, manifest or lockfile. Directory and migration handling were renumbered after the merge: upstream owns the released p0004 (stable Artifact handles), the frozen current business baseline is p0005 (adding bootstrap receipts) and the directory expansion is p0006. Merge-head directory/migration focused tests: 253 passed, 114 skipped (opt-in real seekDB/OceanBase cases); directory core 45 passed; DSH plugin 329 passed; docs build 931 pages with internal links verified. Current head 5e3aa659 merges upstream 0867144c. Existing backend regressions do not establish seekDB/OceanBase directory support. Maintainer review and design acceptance are still required.

AI usage statement

OpenAI Codex assisted implementation, local source/contract review and validation. Local tests, real network/package evidence and unavailable external-backend checks are reported separately; no independent or hosted review pass is claimed.

[wutongyuonce]

@Teingi CI update: the previous dsh-package failure was caused by a stale checked-in bundle. This is fixed in 07375594, and the full DSH job now passes, including the real runtime checks. The conflicts with current master are also resolved.

23 checks are now passing, including Python 3.11–3.14 and OceanBase acceptance. The only remaining failure is Acceptance (sqlite). Its uploaded exception evidence shows that the acceptance, LoCoMo, and fail-fast batches all stopped during agent dependency installation, before scenario execution:

Failed to fetch: https://pypi.org/simple/inquirer-textual/
HTTP status server error (503 Service Unavailable)

The download failed after three retries. This remaining failure is an external dependency-download error, not a pagination assertion failure; it does not establish a SQLite acceptance pass yet.

Could a maintainer please rerun the failed SQLite acceptance job? My attempt to rerun it was rejected with HTTP 403 due to repository permissions.

[Zxf-xufeng]

Minimize additional tables and justify their responsibilities.
This PR introduces pc_memory_entry_directory, pc_memory_tag_generations, and pc_memory_query_index_schema. Please evaluate whether extending existing tables can reduce this footprint. For example, the tag generation could potentially be stored on the owning Artifact head, while migration state should be reviewed alongside the unified migration design.
A separate directory projection may still be justified for efficient historical pagination. Please document why existing tables cannot support it through appropriate extensions. Any alternative must preserve revision-pinned traversal, deactivation/reactivation history, compaction behavior, and bounded reads. Moving an unbounded history into a JSON field would require performance evidence before being considered equivalent.

[Zxf-xufeng]

Align schema changes and historical backfills with PR #1771.
#1771 proposes a unified database migration process. This PR currently adds startup table creation and marker writes, plus a separate memory-query-migrate command that independently creates tables, backfills data, and publishes readiness. Its standalone verify also updates the database.
Please separate schema migration, projection backfill, and readiness verification according to that proposed process:

• Deliver schema changes through versioned migrations, with historical projection reconstruction as an explicit, idempotent data task.
• Keep ordinary startup and inspection commands read-only for existing databases; publish projection readiness during explicit migration execution.
• Use the unified execution boundary for maintenance, locking across batches, backup choices, and final verification.
• Distinguish schema readiness from projection readiness. Legacy APIs may remain available during an optional backfill only after their required schema has been migrated and validated.
• Cover interrupted/resumed backfills and verification failures, including a previously complete projection that subsequently fails verification.
  docs(rfc): define unified versioned database migrations #1771 is still a proposal, so this does not necessarily require waiting for the framework to ship. If this PR lands before framework enablement, please document its baseline adoption path. If it remains unmerged when the framework is enabled, it should follow the unified migration process rather than retain an independent migration authority.

[PsiACE]

@wutongyuonce Thanks for the implementation, tests, and scale validation in this PR. We are reviewing unified database migrations (#1771) and the Memory redesign (#1809, dependent on #1803). These designs would affect the directory projection, revision cursors, and migration approach used here.

We therefore plan to close this PR for now and keep the pagination requirement in #1656 open until those designs are settled. We'd welcome your continued involvement, and the work you've done provides a useful reference for the next implementation.

[hidb4ai]

please update & resolve conflicts

[hidb4ai]

please update & resolve conflicts

[hidb4ai]

please update & resolve conflicts

[wutongyuonce]

@Zxf-xufeng @PsiACE @hidb4ai Conflicts with master are resolved and the PR is mergeable again (5e3aa659, merges upstream 0867144c).

Since your review this PR was refactored onto the existing server db-migrate entry point: the standalone memory-query migration command and its separate migration authority are gone. The optional directory target now delivers schema expansion and the current-head projection backfill as separate steps under one maintenance boundary, ordinary startup and inspection stay read-only for existing databases, and startup remains blocked until recorded maintenance completes. Upstream now owns the released p0004 (stable Artifact handles), so the target adopts the released schema as p0005 (plus bootstrap receipts) and adds only Scope generation and head kind as p0006. No pagination tables were added.

Merge-head evidence: full pytest 5,380 passed / 429 skipped, directory+migration focused 253 passed, directory core 45 passed, DSH 329 passed, make check and the docs build (931 pages) pass.

[hidb4ai]

please update & resolve conflicts

[wutongyuonce]

@hidb4ai resolved

[hidb4ai]

@wutongyuonce Thank you for the substantial work on this PR, especially the compact directory contract, cursor validation, migration integration, and test coverage. We also recognize that the implementation has moved onto the unified migration process and no longer introduces separate pagination tables; the earlier concerns about a standalone migration authority do not describe the current implementation.

With #1882 now merged, we would like to close this PR for now and revisit the directory capability when there is a concrete consumer and a clear backend support plan.

The two features serve different purposes. #1882 provides relevance-based retrieval of Atomic Memory through the shared Artifact search contract. This PR provides a lightweight, filtered, paginated inventory of memory identities and lifecycle metadata. Search does not replace complete directory traversal, so the underlying directory requirement remains valid. However, the directory is not a prerequisite for the primary write → search → exact-read workflow that we want to prioritize now.

Our reasons for deferring this implementation are:

• The immediate product need needs to be clearer. Existing list and exact-read APIs can support basic browsing. Before adding a dedicated directory contract, we would like to identify the actual UI or client that needs it, the expected collection size, and the measurable limitation of the existing path. A compact, bounded directory may be the right solution, but it should be driven by that use case.
• The backend coverage does not yet match our main deployment needs. The directory currently supports a restricted set of SQLite configurations, while OceanBase is an important deployment backend for us. Passing normal OceanBase memory/search tests does not establish support for the new directory operation. We would like an explicit plan for the backend coverage required by its intended consumer.
• The ongoing maintenance cost needs to be justified by the benefit. Even without new directory tables, the feature adds schema fields, backfill and migration work, write-path generation maintenance, and cursor consistency/invalidation rules. These are reasonable mechanisms for a demonstrated bounded-browsing requirement, but they introduce persistent complexity that the core retrieval workflow does not currently require.

If you would like to continue this work, a useful next step would be to define a small, end-to-end browsing use case, document why the existing list/search APIs cannot meet its bounds, and evaluate the implementation against the code now merged in #1882. Please reuse the existing query, filtering, authorization, and persistence boundaries where their contracts fit, and validate the combined implementation on the intended deployment backend. The directory's restart-on-mutation behavior should also be explicit to its consumer; it should not be presented as a snapshot-complete export during ongoing writes.

We are closing this as a scope and prioritization decision, not because directory pagination has no value or because #1882 implements the same feature. The pagination requirement in #1656 remains a separate item to evaluate.

You are very welcome to reopen this PR at any time to continue the discussion or bring back a focused implementation with a concrete consumer and backend plan. Thank you again for your contribution and for working through the reviews and updates.