feat(skill-memory): per-skill cross-session recall + historian auto-extraction

[iceteaSA]

Summary

Adds skill-memory — per-skill "motor memory" that gives a skill cross-session recall of its own hard-won lessons (gotchas, discoveries, fixes, workflow steps). When a skill's SKILL.md declares skill-memory: { enabled: true }, accumulated notes for that skill surface automatically in a <skill-memory> block appended to the skill tool's result on every load — and, with the historian extension, the historian writes those notes automatically during compaction (no agent action required).

It's fully opt-in per skill and cache-safe by construction: the block rides the tool-result tail (conversation), never the cached system/m[0] prefix, so it can't bust the prompt cache.

How it works

• Transparent recall — three-hook augmentation: tool.definition advertises an intent param; tool.execute.before stashes the intent (bounded TTL); the after-hook parses the skill's Base directory, reads its SKILL.md frontmatter, and formats the recall block. Lands in the tool RESULT (cache-safe).
• Write paths — ctx_skill_note (agent-authored) and the historian (auto-extracted). Both dedup on a normalized hash.
• Intent-scoped ranking — a recall cascade: model-matched embeddings → cosine blend over intent_embedding + delta_embedding (relevance/recency/hit weights tunable per skill via ranking_* frontmatter); FTS5 fallback over a content-linked skill_memory_fts vtable; flat recency×hit fallback.
• Historian auto-extraction — the historian sees TC: skill(<name>) markers in its chunk and emits a <skill_observations> block; both the OpenCode and Pi runners promote those post-commit as global notes (project_identity='*', source_type='historian'), recallable from any project.

Review units (4 commits)

The branch is organized into four coherent, reviewable phases:

1. P1 — transparent per-skill recall: skill_memory table migration, the three-hook augmentation, flat recall + storage, ctx_skill_note/ctx_skill_recall tools, opt-in distill-skill-memory dreamer task, TUI/ctx-status stats, docs.
2. P2 — embeddings + intent-scoped recall: delta_embedding + recall_count columns + skill_memory_fts FTS5 vtable; embed-on-write; the cosine/FTS recall cascade; a programmatic (no-LLM) reembed pre-step.
3. P3a — historian-extraction foundation: the TC: skill(<name>) marker keystone; origin_project + source_type columns; global-tier notes unified under project_identity='*' (collision-merge); partitionKey routing.
4. P3b — historian auto-extraction pipeline: prompt emits <skill_observations>, parser, validated-result threading, both runners promote via the shared promoteSkillObservations helper, plus an initializeDatabase self-heal net (re-creates skill_memory + ensureColumn so an upgraded DB recovers even if a migration row is lost).

Schema / migrations

Three migrations (numbered after the current master ceiling): skill_memory table, then delta_embedding/recall_count/FTS, then origin_project/source_type/* unification. LATEST_SUPPORTED_VERSION bumped in lockstep (the schema-version-fence test enforces it). Migration bodies are ensureColumn/IF NOT EXISTS idempotent.

Testing

Full plugin + Pi suites pass (one unrelated pre-existing full-suite ordering flake in tui-config.test.ts that passes in isolation), tsc clean both packages, lint clean, build produces all bundles. Dedicated coverage for the migrations (coexistence + fence), recall rungs, the tools, FTS triggers/backfill, the '*' collision-merge, and the historian promotion path on both runners. Verified working live: the historian auto-extraction writes genuine source_type='historian' notes, embeddings populate, and the read-side recall_count increments on surfacing.

Notes for reviewers

• Migration version numbers are placeholders relative to this fork's base — happy to renumber to whatever slots are free at merge time.
• The four commits are independently meaningful; if you'd prefer this as stacked PRs (P1 / P2 / historian), I can split it.

---

^{Need help on this PR? Tag /codesmith with what you need. Autofix is disabled.}

---

Summary by cubic

Adds per-skill cross-session recall and historian auto-extraction. Previously skills had no recall; now opted-in skills append a cache-safe <skill-memory> block on each load, and the historian writes global notes so skills remember gotchas, discoveries, fixes, and workflows without busting the prompt cache.

• Transparent recall: the skill tool exposes an optional intent; hooks stash it and append a rendered recall block. Ranking cascades from cosine over on-the-fly intent + stored delta_embedding, to FTS5, then recency×hit. Reads union the skill’s project partition with the global '*' tier; reads increment recall_count. Token budgeting counts per-note framing; pinned budget clamps to min(max_pinned_tokens, max_tokens).
• Write/recall tools: ctx_skill_note dedups by hash and cosine, embeds on write, and rejects writes when a skill hasn’t enabled memory. ctx_skill_recall renders notes on demand. Both are OpenCode-only; Pi does not register them.
• Historian auto-extraction: the historian emits <skill_observations>; both OpenCode and Pi promote them as global '*' notes (source_type='historian') via a shared helper, gated to avoid double-emits.
• Provenance and safety: accepts plain paths and file://; Windows path normalization; name-based fallback when the trailing provenance line is truncated. Intent stash is keyed by sessionId:callId with TTL and per-session pruning. The skill tool’s model schema advertises the intent param via jsonSchema. ctx_skill_note now fails early when a skill hasn’t opted in.
• UX and surface: async executeStatus and the TUI show per-project skill-memory totals. The system prompt adds skill-memory guidance. The A1 golden and prompt-surface budget fixture were re-measured to include ctx_skill_note and ctx_skill_recall under the existing policy.
• Dreamer: adds opt-in distill-skill-memory; a pre-step re-embeds NULL/stale vectors, then emits a read-only corpus health report.

Rollout

• Fork-lane migrations v10000–v10002 add skill_memory, embeddings + FTS, and historian provenance/global '*' unification. initializeDatabase self-heals (creates table/columns, rebuilds FTS) and runForkMigrations runs on DB open.
• Required action: enable per skill by adding skill-memory: { enabled: true } to SKILL.md. Optional: schedule the distill-skill-memory dreamer task.

^{Written for commit 5c5fb3b. Summary will update on new commits.}

Greptile Summary

This PR introduces skill-memory — per-skill cross-session recall that appends a <skill-memory> block to a skill's tool result when its SKILL.md declares skill-memory: { enabled: true }. The historian auto-extraction pipeline writes global notes under project_identity='*' so lessons accumulate across projects without busting the prompt cache.

• Three fork-lane migrations (v10000–v10002) add skill_memory, embeddings+FTS, and historian provenance columns; initializeDatabase self-heals on a DB where migrations ran partially.
• A four-rung recall cascade (cosine over intent+delta embeddings → FTS5 → flat recency×hit) is wired into tool.execute.before/after hooks; two new tools (ctx_skill_note, ctx_skill_recall) expose the write and explicit-read paths to the agent.
• Both the OpenCode compartment runner and the Pi historian runner promote <skill_observations> blocks as global '*' notes; the recallPartitionPredicate union ensures project-local skills still surface those global historian rows on recall.

Confidence Score: 5/5

• This PR is safe to merge. The core recall, storage, and historian promotion paths are well-implemented and previously flagged bugs have all been addressed.
• The three bugs identified in the previous review round (session-bleed on intent stash, FTS rebuild gap in self-heal, orphaned historian notes for project-local skills) were all fixed. The remaining findings are a misleading parameter name on stashIntent and a redundant double-call inside searchSkillMemoryFts — neither affects correctness at runtime.
• No files require special attention beyond the stashIntent parameter naming in hook-handlers.ts, which is a latent readability concern rather than an active defect.

Important Files Changed

Filename	Overview
packages/plugin/src/hooks/magic-context/hook-handlers.ts	Adds the intent stash map helpers (stashIntent, getAndDeleteIntent, pruneIntentsForSession, intentKey) and the three-hook plumbing (createToolExecuteBeforeHook, maybeInjectSkillMemory). The session-composite key pattern is implemented correctly in production, but stashIntent's callId parameter name is misleading — it must receive a pre-built intentKey() composite; calling it with a bare callId silently bypasses the session-prune mechanism.
packages/plugin/src/features/magic-context/skill-memory/storage.ts	New storage layer for skill_memory. The recallPartitionPredicate helper correctly unions the skill's own partition with the global '*' partition so historian-promoted notes are surfaced regardless of skill tier. searchSkillMemoryFts calls recallPartitionPredicate twice with identical arguments; a single destructured call would be cleaner.
packages/plugin/src/features/magic-context/skill-memory/recall.ts	Four-rung recall cascade (cosine → FTS5 → flat recency×hit) is well-structured. Token budgeting via budgetFill clamps the pinned sub-budget to min(max_pinned_tokens, max_tokens) correctly. sanitizeSkillIntentForFts properly quotes FTS5 tokens. Top-level catch now logs via log() rather than silently swallowing errors.
packages/plugin/src/features/magic-context/skill-memory/promote.ts	Historian extraction promotion: always writes to tier='global', project_identity='*'. Hash dedup bumps hit_count instead of inserting duplicates. Per-observation error isolation ensures one bad observation never blocks the rest. Clean implementation.
packages/plugin/src/features/magic-context/skill-memory/frontmatter.ts	Custom minimal YAML parser for the skill-memory: block. Correctly handles both inline flow-mapping and indented block forms, inline comments, BOM, and CRLF. Returns null (inert) for malformed or absent configs rather than throwing. No dependencies on a full YAML library.
packages/plugin/src/features/magic-context/skill-memory/provenance.ts	Skill path/tier/source resolution. Accepts both file:// URLs (legacy opencode) and plain paths (current). The name-based fallback resolveSkillPathByName correctly skips project-tier candidates when projectDirectory is null to avoid poisoning the registry with wrong-dir guesses. Ancestor walk stops at .git boundary, $HOME, and filesystem root.
packages/plugin/src/features/magic-context/fork-migrations.ts	Three fork-lane migrations (v10000–v10002) add skill_memory, embeddings+FTS, and historian provenance columns. Migration bodies use IF NOT EXISTS / ensureColumn for idempotency. Kept in a separate runForkMigrations() so upstream's migration fence and version tests are unaffected.
packages/plugin/src/features/magic-context/storage-db.ts	Self-heal additions to initializeDatabase: creates skill_memory table, FTS virtual table, and three content-table triggers with IF NOT EXISTS. The guarded FTS rebuild (ftsCount === 0 && rowCount > 0) correctly backfills pre-existing rows without re-indexing on every boot. runForkMigrations is called from both openDatabase and openDatabaseAsync.
packages/plugin/src/hooks/magic-context/compartment-parser.ts	Adds <skill_observations> block parsing. The SKILL_OBS_ITEM_REGEX correctly requires *-prefixed lines in `name
packages/plugin/src/tools/ctx-skill-note/tools.ts	Note creation tool. Guards against disabled skill-memory before any DB work. Hash dedup first, then cosine dedup bounded to 200 candidates. Handles the concurrent-insert race (UNIQUE constraint) by bumping hit_count on null return from insertSkillMemoryNote. Correctly computes project identity from toolContext.directory.
packages/plugin/src/tools/ctx-skill-recall/tools.ts	Explicit recall tool with registry-first then disk-fallback resolution. Guards against disabled frontmatter before recall. The test-injection path defaults tier to "global", which may miss project-tier edge cases in tests but doesn't affect production. Returns distinct messages for "skill not found", "memory disabled", and "no notes yet".

Sequence Diagram

sequenceDiagram
    participant Agent
    participant SkillTool as skill tool
    participant BeforeHook as tool.execute.before
    participant AfterHook as tool.execute.after
    participant Registry as SkillLoadRegistry
    participant DB as skill_memory DB
    participant Recall as recallSkillMemoryBlock

    Agent->>SkillTool: skill(name, intent)
    SkillTool->>BeforeHook: fires pre-validation
    BeforeHook->>BeforeHook: stashIntent(map, sessionId:callId, intent)

    SkillTool-->>AfterHook: output with "Base directory for this skill:"
    AfterHook->>AfterHook: parseSkillProvenance(output)
    AfterHook->>AfterHook: parseFrontmatterConfig(SKILL.md)
    AfterHook->>Registry: set(sessionId:skillId, provenance + frontmatterConfig)
    AfterHook->>AfterHook: getAndDeleteIntent(map, sessionId:callId)
    AfterHook->>Recall: "recallSkillMemoryBlock(db, {skill, intent, scope, projectIdentity})"

    alt has intent + embeddings
        Recall->>DB: getRankingCandidates (200)
        Recall->>Recall: rankRung1(cosine over intent+delta vectors)
        Recall->>DB: getPinnedNotes
        Recall->>Recall: unionPinnedFirst + budgetFill
    else FTS fallback
        Recall->>DB: searchSkillMemoryFts(sanitized intent)
    else flat fallback
        Recall->>DB: getSkillMemoryNotes(recency×hit)
    end

    Recall->>DB: bumpRecallCountByIds(surfaced note ids)
    Recall-->>AfterHook: "<skill-memory> XML block"
    AfterHook->>AfterHook: "output.output += block"
    AfterHook-->>Agent: augmented skill tool result

    Note over Agent,DB: Agent may call ctx_skill_note to write a note
    Agent->>DB: ctx_skill_note(skill, intent, kind, delta)
    DB-->>Agent: "Skill note saved (id=…)"

    Note over DB: Historian compaction (async)
    AfterHook->>DB: "promoteSkillObservations(global, '*', observations)"

Loading

_{Reviews (38): Last reviewed commit: "chore(skill-memory): rebase onto upstrea..." | Re-trigger Greptile}

[cubic-dev-ai]

11 issues found across 78 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/plugin/src/features/magic-context/dreamer/task-executor.ts">

<violation number="1" location="packages/plugin/src/features/magic-context/dreamer/task-executor.ts:419">
P2: Re-embed pre-step errors are suppressed, allowing successful task completion reporting despite a failed prerequisite data maintenance step.</violation>
</file>

<sub>Note: This PR contains a large number of files. cubic only reviews up to 40 files per PR, so some files may not have been reviewed. cubic prioritizes the most important files to review.
On a pro plan you can use ultrareview for larger PRs.

Re-trigger cubic</sub>

[cubic-dev-ai]

4 issues found across 15 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/plugin/src/features/magic-context/dreamer/task-executor.ts">

<violation number="1" location="packages/plugin/src/features/magic-context/dreamer/task-executor.ts:419">
P2: Re-embed pre-step errors are suppressed, allowing successful task completion reporting despite a failed prerequisite data maintenance step.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

4 issues found across 78 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/plugin/src/features/magic-context/dreamer/task-executor.ts">

<violation number="1" location="packages/plugin/src/features/magic-context/dreamer/task-executor.ts:419">
P2: Re-embed pre-step errors are suppressed, allowing successful task completion reporting despite a failed prerequisite data maintenance step.</violation>
</file>

<sub>Note: This PR contains a large number of files. cubic only reviews up to 40 files per PR, so some files may not have been reviewed. cubic prioritizes the most important files to review.
On a pro plan you can use ultrareview for larger PRs.

Re-trigger cubic</sub>

[cubic-dev-ai]

1 issue found across 2 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[alfonso-magic-context]

Thank you for the sustained work on this and for keeping it rebased — the persistence deserves a straight answer rather than another quiet month, so here it is: we're not taking a skill-memory subsystem into Magic Context. Cross-session skill recall is being solved in our stack by a different mechanism (retrieval-on-demand skill documents with their own authoring pipeline, outside the plugin's memory system), and the historian auto-extraction lane here would overlap it while adding a second promotion pipeline to a memory system we're actively simplifying — see the design discussion in #335 for the direction. Closing as not planned. That's a verdict on fit, not on the work: the implementation quality across your PRs has been consistently high, and the extractions we've merged from them are credited in the history.