Repository navigation
Conversation
…, hinotes connector, hidock-mcp v2, knowledge-graph expansion Consolidating everything that was sitting uncommitted in this checkout before forking away from upstream. No behaviour intentionally changed by this commit beyond what was already running locally.
Arm A (current prompt + llama3.2) vs arm C (grounded prompt + gemma3:12b). Arm B rejected (few-shot bleed + hallucination). Includes raw per-meeting output, harness scripts, and all three prompts. Health Concerns meeting confirmed clean under arm C (no fabrication).
…nded prompt, gemma3:12b extraction model, within-result dedup B1 (promote): new promoteExtractionToFirstClassTables() in database.ts writes extractGraphFromTranscript's decisions/action_items into the first-class decisions/action_items tables the hidock-mcp server reads, keyed by knowledge_capture_id, honest-subset only (NULL not fabricated), content-deduped. Wired into ingestFromDbTranscripts inside the existing ingest transaction. Extraction quality (eval/extraction-prompt-2026-09): - extract.ts buildPrompt: grounded, rules-not-exemplars prompt. Fixes 3B-model fragment output; the explicit anti-hallucination/grounding rule + no verbatim exemplars prevent the fabrication + few-shot bleed seen on the exemplar variant. - parseExtractionOutput: within-result de-duplication so a model repeating the same item 3x-5x in one meeting cannot become duplicate graph nodes or DB rows. - Dedicated chat.extractionOllamaModel (default gemma3:12b) + getExtractionProviderConfig(); extraction uses the stronger local model WITHOUT changing chat/value-classification. Tests: extract.test.ts 9/9 (incl. dedup), ai-provider-config.test.ts 10/10 (incl. override), promote-extraction.test.ts. Electron typecheck clean.
…e-ingest dry-run script completeWithRetry() retries ONLY transient fetch/network failures (observed on 29k-67k transcripts against local Ollama) with bounded attempts + backoff; parse failures and non-transient errors are not retried. Wraps the 3 extraction llm seams. Needed for the 19-recording re-ingest and the eventual 790 backfill. reingest-dryrun.mjs: read-only, derives scope at runtime (all transcripts with a graph_ingested_transcripts marker) and reports recording IDs + removal counts.
…un supported)
Exposes a new channel that, for every recording with a live
graph_ingested_transcripts marker (scope derived at runtime, not hardcoded):
1. calls the tested removeRecordingProvenanceCore(rid) — protects shared
edges, GCs orphan nodes by type, scrubs merge-journal snapshots, removes
markers;
2. deletes that capture's first-class decisions/action_items rows (B1 output,
which the provenance core predates and does not touch).
Per-recording removal+B1-delete run in one transaction. dryRun:true writes
nothing and reports the same scope + counts. Re-extraction stays a separate
graph:ingestAll call so the cleaned state is verifiable in between. Needed
because removeRecordingProvenanceCore was previously only reachable via
deleteRecordingCascade (which destroys the source recording). database import is
lazy inside the handler (avoids dragging config.ts app.getPath() at module load,
which the handler test's electron mock does not provide). Handler test 20/20.
…o-capture, error)
Adds dedicated tests for the destructive re-ingest removal handler:
- dry-run writes nothing (no run/runInTransaction) but reports scope + counts
- real-run deletes B1 decisions/action_items AND calls removeRecordingProvenanceCore
per in-scope recording, each in its own transaction
- a recording with no matching capture_id does not error and skips the B1 delete
- removal throwing returns { success:false, error }
Bumps the channel-registration assertion to 8 and includes graph:reingestRecordings.
Handler test: 24/24.
Each decision/action_item/risk/next_step is now tagged category:'work'|'personal' by the grounded prompt (health/medical/family/finances/etc. -> personal; 'if unsure, tag personal' privacy-safe default). parseExtractionOutput drops any item not explicitly 'work' (personal, missing, unknown, malformed) BEFORE dedup and before the result reaches ingestExtraction (graph_nodes) OR the B1 promote (relational decisions/action_items). Filtering only at the B1 promote would relocate the leak: graph_nodes are queryable via hidock_search/topAttendees/ meetingGraph too. Dropping at the parse boundary is the only version that actually contains it — both consumers simply never receive personal items. Tests 13/13 (4 new: personal dropped from all 4 types, missing->dropped, unknown->dropped/case-insensitive-work-kept, fully-personal->empty). Validated on a synthetic mixed work/personal meeting: work items kept, all personal items dropped, no personal content leaked into either consumer. Eval details are recorded outside version control (contained real personal content and are intentionally not committed).
…pt fetch drops) Non-streaming stream:false holds one long HTTP response open for the entire generation; on large transcripts (27k+ chars on gemma3:12b) this intermittently drops with 'TypeError: fetch failed' before the body arrives — observed failing 4x on the 27k Health Concerns transcript. stream:true keeps the connection fed with incremental NDJSON token fragments, which is robust on long local generations; complete() reassembles the full text and preserves the same string return contract + error semantics (HTTP failure, embedded stream error, empty output). Handles fragments split across chunk boundaries and a newline-less tail. Verified live: the same 27k transcript that dropped 4x non-streaming completed in 37.7s first try with streaming. Tests 6/6 (reassembly, split-boundary, tail-flush, stream-error surfacing, HTTP failure). This is the pre-req for the 19-recording re-ingest (and the 790 backfill) — avoids a retry-storm on the large transcripts.
Personal-content boundary (fail-closed parse-time drop), typed malformed-extraction handling, recording/capture eligibility guard with per-attempt recheck, scoped re-ingestion handler (discovery manifest + Scope_Digest + validated scoped deletion + whole-batch/journaled runs), ingestion_run provenance + backward-compatible marker migration + version/hash staleness + atomic current-marking, model isolation (getExtractionProviderConfig), and hardened transient retry (typed transport classifier + bounded exponential backoff/jitter, cancellation-aware). Fixes the recording-provenance test-harness better-sqlite3 ABI binding (createRequire the database pkg's copy). Adds synthetic personal-content eval. Design-only Littlebird/large-batch artifacts live in the work-brain spec, not here. Note: scoped/journaled re-ingestion runners are exported + tested but not yet wired to a live invocation path (the extracted_from-scoped deleter is for the future 790-item backfill); today's live cleanup used the existing graph:reingestRecordings handler.
Implements the design-only Littlebird integration surfaces from littlebird-adapter-interface.md and notes/littlebird-cross-source-dedup.md: canonical capture mapping with a fail-closed classifier boundary, and dedup evaluation across identifier, timestamp/participant-overlap, content-hash, calendar-anchor, and explicit-equivalence rules. No live Littlebird connection, network call, or auth flow is included — matches the spec's stated "not authorised" boundary. Adds synthetic-fixture tests closing out task 12.6 (11 adapter + 15 dedup tests). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W3T66EZfvq3aX47UZKcxmM
…captures (schema v55) Phase 2 of the Littlebird relay: Work Brain's DB previously had no first-class home for risk/question items (only decisions/action_items existed), so this adds `risks` and `questions` tables mirroring their shape (content, extraction metadata, status/severity, timestamps, FK to knowledge_captures). Also adds source_connector_id/source_ref to knowledge_captures so a connector-pushed capture (e.g. from the Littlebird relay) can be deduped the same way artifact-service already dedupes artifacts, via a partial unique-lookup index on (source_connector_id, source_ref). Verified the fresh-install SCHEMA and the v55 MIGRATIONS block separately against a real SQLite engine (schema applies cleanly, migration path from a pre-v55 DB adds the columns/tables idempotently, insert round-trip works) — the in-repo vitest suite couldn't run in this environment due to a pre-existing rollup native-binary mismatch unrelated to this change. No live data written by this commit — the actual DB migration only runs the next time the app boots, via the existing DatabaseEngine migration runner (which also takes its own pre-migration backup). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W3T66EZfvq3aX47UZKcxmM
…ables (v55) Wires the risks/questions schema (previous commit) into the connector ingestion pipeline: a connector that emits a SourceItem of kind 'decision' | 'action_item' | 'risk' | 'question' with a structured `entity` set (new ExternalDecision/ExternalActionItem/ExternalRisk/ ExternalQuestion types in @hidock/connectors) now gets upserted into the matching first-class table via the new database.ts upsertConnectorKnowledgeItem(), instead of falling through to the generic artifact-service importer. - database.ts: upsertConnectorKnowledgeItem() — keyed on (source_connector_id, source_ref) via knowledge_captures, so a re-sync of the same item updates the existing capture + child row rather than duplicating. Verified every INSERT/UPDATE statement (all four kinds, including the NULL-optional-field -> CHECK-constraint-default path) directly against real SQLite. - ingestion.ts: routing branch + pure sourceItemToConnectorKnowledgeItem mapper (exported for testing, same pattern as externalMeetingToRow). - registry.ts: threads the new IngestionOutcome.knowledgeItems counter through syncContainer/syncNow's accumulation. - Extended the existing ingestion.test.ts (fakeDeps, mocks, routing assertions) and added coverage for the new routing branch and mapper. Couldn't run the vitest suite in this sandbox — it hits a pre-existing rollup native-binary mismatch (linux-arm64 binary missing) unrelated to this change, most likely because this sandbox bridges into a Linux VM rather than your Mac's own arm64 darwin node_modules. Worth running `npx vitest run` for these two files on your machine to confirm before merging. Still not built: the Supabase staging table and the real Vercel relay tools (log_decision/log_risk/etc.) that will actually call upsertConnectorKnowledgeItem via this routing — this commit only wires the receiving end. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W3T66EZfvq3aX47UZKcxmM
… Brain Registers a new zero-auth connector (littlebird-relay) that polls the relay_staged_items table in Supabase (written to by the workbrain-relay Vercel MCP server a Littlebird Routine calls) and routes staged decisions/action items/risks/questions through the existing upsertConnectorKnowledgeItem path (schema v55), exactly like any other source. Delivery is at-least-once; ingestion is idempotent on (connector_id, source_ref) so a redelivered row is a safe no-op. Registered in initConnectors() with a 5-minute scheduled sync, mirroring the HiNotes connector's pattern. Config is just a Supabase project URL + its Secret key (required — the staging table has RLS enabled with no policies, so only the Secret key can read it). Validated with a standalone smoke test against a faked fetch (mapping, cursor advancement, PATCH shape) since vitest cannot run in this sandbox (pre-existing @rollup/rollup-linux-arm64-gnu platform mismatch unrelated to this change) — please run `npx vitest run` on your Mac to confirm before merging. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W3T66EZfvq3aX47UZKcxmM
Adds confidence vocabulary and provenance to the four first-class knowledge tables (decisions/action_items/risks/questions) ahead of wiring them to Empirical, per Work Brain architecture-discovery.md section 6 item 2: - Schema v56: `certainty` (confirmed/proposed/assumed/superseded, defaults to confirmed) and `evidence` (source quote/reference) on all four tables; `superseded_by` FK on decisions only, with a partial index. - Migration 56 adds the columns idempotently via the existing getTableColumns guard for pre-v56 databases. - KNOWLEDGE_CERTAINTY_VALUES + normalizeCertainty() validate/coerce incoming values (unknown input falls back to 'confirmed'). - ConnectorKnowledgeItem and packages/connectors' External* types carry the new optional fields end-to-end from connector source items through upsertConnectorKnowledgeItem's SQL. - ingestion.ts passes certainty/evidence/supersededBy through sourceItemToConnectorKnowledgeItem for all four kinds. - Tests added for certainty/evidence passthrough and null defaults. Validated directly against SQLite (fresh-install schema, simulated pre-v56 migration path, all INSERT/UPDATE variants) and via `tsc --noEmit` on database.ts, since vitest cannot run in this sandbox. Needs `npm run build` in packages/connectors and `npx vitest run` on a real machine before merging. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W3T66EZfvq3aX47UZKcxmM
…st checks - resolveClaudeCommand used POSIX path.join on Windows-style trusted-root paths; now uses win32.join on win32 - validateTargetDir couldn't canonicalize a protected path that doesn't exist yet, leaving an unresolved-symlink bypass in the ancestor check Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W3T66EZfvq3aX47UZKcxmM
…k suite - reanalyze-exclusion.test.ts: mock brain-registry, not just brains helpers (stale after BrainRouter refactor) - ingestion.test.ts: expected object missing schema v56 certainty/evidence fields - handover-service.test.ts: assert against realpath, not raw mkdtemp path (macOS /var -> /private/var) - download-service-session-c5.test.ts: existsSync mock made path-specific so queueDownloads' own disk-reconciliation logic isn't fooled into treating queued files as already synced - meeting-recorder database.test.ts: full rewrite off a dead sql.js mock, now exercises the real better-sqlite3-backed DatabaseEngine directly Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W3T66EZfvq3aX47UZKcxmM
|
Important Review skippedToo many files! This PR contains 108 files, which is 8 over the limit of 100. To get a review, reduce the PR to 100 files or fewer by splitting it into smaller PRs or changing its base branch. Upgrade to a paid plan to raise the limit. This review couldn't start because sufficient usage credits or metered capacity aren't available. Add credits or update usage-based reviews in the billing tab, then retry. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Team Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (108)
You can disable this status message by setting the Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Pull Request
Description
Brief description of the changes in this PR.
Type of Change
Related Issues
Fixes #(issue number)
Closes #(issue number)
Related to #(issue number)
Changes Made
Testing
Desktop Application
Web Application
Device Testing
Screenshots (if applicable)
Add screenshots to help explain your changes.
Checklist
Breaking Changes
If this PR introduces breaking changes, please describe them here and provide migration instructions.
Additional Notes
Any additional information that reviewers should know about this PR.