Skip to content
This repository was archived by the owner on Sep 17, 2026. It is now read-only.

Work brain relay schema - #83

Closed
kelly1175 wants to merge 16 commits into
sgeraldes:mainfrom
kelly1175:work-brain-relay-schema
Closed

kelly1175 wants to merge 16 commits into
sgeraldes:mainfrom
kelly1175:work-brain-relay-schema

Conversation

@kelly1175

Copy link
Copy Markdown

Pull Request

Description

Brief description of the changes in this PR.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Code refactoring
  • Performance improvement
  • Test improvements

Related Issues

Fixes #(issue number)
Closes #(issue number)
Related to #(issue number)

Changes Made

  • Change 1
  • Change 2
  • Change 3

Testing

Desktop Application

  • Unit tests pass
  • Integration tests pass
  • Manual testing completed
  • Tested on Windows
  • Tested on macOS
  • Tested on Linux

Web Application

  • Unit tests pass
  • Component tests pass
  • E2E tests pass
  • Tested in Chrome
  • Tested in Firefox
  • Tested in Edge

Device Testing

  • Tested with H1 device
  • Tested with H1E device
  • Tested with P1 device
  • Connection/disconnection tested
  • File operations tested

Screenshots (if applicable)

Add screenshots to help explain your changes.

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published

Breaking Changes

If this PR introduces breaking changes, please describe them here and provide migration instructions.

Additional Notes

Any additional information that reviewers should know about this PR.

kelly1175 and others added 16 commits September 3, 2026 13:51
…, hinotes connector, hidock-mcp v2, knowledge-graph expansion

Consolidating everything that was sitting uncommitted in this checkout
before forking away from upstream. No behaviour intentionally changed by
this commit beyond what was already running locally.
Arm A (current prompt + llama3.2) vs arm C (grounded prompt + gemma3:12b).
Arm B rejected (few-shot bleed + hallucination). Includes raw per-meeting
output, harness scripts, and all three prompts. Health Concerns meeting
confirmed clean under arm C (no fabrication).
…nded prompt, gemma3:12b extraction model, within-result dedup

B1 (promote): new promoteExtractionToFirstClassTables() in database.ts writes
extractGraphFromTranscript's decisions/action_items into the first-class
decisions/action_items tables the hidock-mcp server reads, keyed by
knowledge_capture_id, honest-subset only (NULL not fabricated), content-deduped.
Wired into ingestFromDbTranscripts inside the existing ingest transaction.

Extraction quality (eval/extraction-prompt-2026-09):
- extract.ts buildPrompt: grounded, rules-not-exemplars prompt. Fixes 3B-model
  fragment output; the explicit anti-hallucination/grounding rule + no verbatim
  exemplars prevent the fabrication + few-shot bleed seen on the exemplar variant.
- parseExtractionOutput: within-result de-duplication so a model repeating the
  same item 3x-5x in one meeting cannot become duplicate graph nodes or DB rows.
- Dedicated chat.extractionOllamaModel (default gemma3:12b) + getExtractionProviderConfig();
  extraction uses the stronger local model WITHOUT changing chat/value-classification.

Tests: extract.test.ts 9/9 (incl. dedup), ai-provider-config.test.ts 10/10
(incl. override), promote-extraction.test.ts. Electron typecheck clean.
…e-ingest dry-run script

completeWithRetry() retries ONLY transient fetch/network failures (observed on
29k-67k transcripts against local Ollama) with bounded attempts + backoff; parse
failures and non-transient errors are not retried. Wraps the 3 extraction llm
seams. Needed for the 19-recording re-ingest and the eventual 790 backfill.

reingest-dryrun.mjs: read-only, derives scope at runtime (all transcripts with a
graph_ingested_transcripts marker) and reports recording IDs + removal counts.
…un supported)

Exposes a new channel that, for every recording with a live
graph_ingested_transcripts marker (scope derived at runtime, not hardcoded):
  1. calls the tested removeRecordingProvenanceCore(rid) — protects shared
     edges, GCs orphan nodes by type, scrubs merge-journal snapshots, removes
     markers;
  2. deletes that capture's first-class decisions/action_items rows (B1 output,
     which the provenance core predates and does not touch).
Per-recording removal+B1-delete run in one transaction. dryRun:true writes
nothing and reports the same scope + counts. Re-extraction stays a separate
graph:ingestAll call so the cleaned state is verifiable in between. Needed
because removeRecordingProvenanceCore was previously only reachable via
deleteRecordingCascade (which destroys the source recording). database import is
lazy inside the handler (avoids dragging config.ts app.getPath() at module load,
which the handler test's electron mock does not provide). Handler test 20/20.
…o-capture, error)

Adds dedicated tests for the destructive re-ingest removal handler:
- dry-run writes nothing (no run/runInTransaction) but reports scope + counts
- real-run deletes B1 decisions/action_items AND calls removeRecordingProvenanceCore
  per in-scope recording, each in its own transaction
- a recording with no matching capture_id does not error and skips the B1 delete
- removal throwing returns { success:false, error }
Bumps the channel-registration assertion to 8 and includes graph:reingestRecordings.
Handler test: 24/24.
Each decision/action_item/risk/next_step is now tagged category:'work'|'personal'
by the grounded prompt (health/medical/family/finances/etc. -> personal; 'if
unsure, tag personal' privacy-safe default). parseExtractionOutput drops any
item not explicitly 'work' (personal, missing, unknown, malformed) BEFORE dedup
and before the result reaches ingestExtraction (graph_nodes) OR the B1 promote
(relational decisions/action_items). Filtering only at the B1 promote would
relocate the leak: graph_nodes are queryable via hidock_search/topAttendees/
meetingGraph too. Dropping at the parse boundary is the only version that
actually contains it — both consumers simply never receive personal items.
Tests 13/13 (4 new: personal dropped from all 4 types, missing->dropped,
unknown->dropped/case-insensitive-work-kept, fully-personal->empty). Validated
on a synthetic mixed work/personal meeting: work items kept, all personal items
dropped, no personal content leaked into either consumer. Eval details are
recorded outside version control (contained real personal content and are
intentionally not committed).
…pt fetch drops)

Non-streaming stream:false holds one long HTTP response open for the entire
generation; on large transcripts (27k+ chars on gemma3:12b) this intermittently
drops with 'TypeError: fetch failed' before the body arrives — observed failing
4x on the 27k Health Concerns transcript. stream:true keeps the connection fed
with incremental NDJSON token fragments, which is robust on long local
generations; complete() reassembles the full text and preserves the same string
return contract + error semantics (HTTP failure, embedded stream error, empty
output). Handles fragments split across chunk boundaries and a newline-less tail.

Verified live: the same 27k transcript that dropped 4x non-streaming completed
in 37.7s first try with streaming. Tests 6/6 (reassembly, split-boundary,
tail-flush, stream-error surfacing, HTTP failure). This is the pre-req for the
19-recording re-ingest (and the 790 backfill) — avoids a retry-storm on the
large transcripts.
Personal-content boundary (fail-closed parse-time drop), typed malformed-extraction handling, recording/capture eligibility guard with per-attempt recheck, scoped re-ingestion handler (discovery manifest + Scope_Digest + validated scoped deletion + whole-batch/journaled runs), ingestion_run provenance + backward-compatible marker migration + version/hash staleness + atomic current-marking, model isolation (getExtractionProviderConfig), and hardened transient retry (typed transport classifier + bounded exponential backoff/jitter, cancellation-aware). Fixes the recording-provenance test-harness better-sqlite3 ABI binding (createRequire the database pkg's copy). Adds synthetic personal-content eval. Design-only Littlebird/large-batch artifacts live in the work-brain spec, not here.

Note: scoped/journaled re-ingestion runners are exported + tested but not yet wired to a live invocation path (the extracted_from-scoped deleter is for the future 790-item backfill); today's live cleanup used the existing graph:reingestRecordings handler.
Implements the design-only Littlebird integration surfaces from
littlebird-adapter-interface.md and notes/littlebird-cross-source-dedup.md:
canonical capture mapping with a fail-closed classifier boundary, and
dedup evaluation across identifier, timestamp/participant-overlap,
content-hash, calendar-anchor, and explicit-equivalence rules.

No live Littlebird connection, network call, or auth flow is included —
matches the spec's stated "not authorised" boundary. Adds synthetic-fixture
tests closing out task 12.6 (11 adapter + 15 dedup tests).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W3T66EZfvq3aX47UZKcxmM
…captures (schema v55)

Phase 2 of the Littlebird relay: Work Brain's DB previously had no
first-class home for risk/question items (only decisions/action_items
existed), so this adds `risks` and `questions` tables mirroring their
shape (content, extraction metadata, status/severity, timestamps, FK to
knowledge_captures).

Also adds source_connector_id/source_ref to knowledge_captures so a
connector-pushed capture (e.g. from the Littlebird relay) can be deduped
the same way artifact-service already dedupes artifacts, via a partial
unique-lookup index on (source_connector_id, source_ref).

Verified the fresh-install SCHEMA and the v55 MIGRATIONS block separately
against a real SQLite engine (schema applies cleanly, migration path from
a pre-v55 DB adds the columns/tables idempotently, insert round-trip
works) — the in-repo vitest suite couldn't run in this environment due to
a pre-existing rollup native-binary mismatch unrelated to this change.

No live data written by this commit — the actual DB migration only runs
the next time the app boots, via the existing DatabaseEngine migration
runner (which also takes its own pre-migration backup).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W3T66EZfvq3aX47UZKcxmM
…ables (v55)

Wires the risks/questions schema (previous commit) into the connector
ingestion pipeline: a connector that emits a SourceItem of kind
'decision' | 'action_item' | 'risk' | 'question' with a structured
`entity` set (new ExternalDecision/ExternalActionItem/ExternalRisk/
ExternalQuestion types in @hidock/connectors) now gets upserted into the
matching first-class table via the new database.ts
upsertConnectorKnowledgeItem(), instead of falling through to the
generic artifact-service importer.

- database.ts: upsertConnectorKnowledgeItem() — keyed on
  (source_connector_id, source_ref) via knowledge_captures, so a re-sync
  of the same item updates the existing capture + child row rather than
  duplicating. Verified every INSERT/UPDATE statement (all four kinds,
  including the NULL-optional-field -> CHECK-constraint-default path)
  directly against real SQLite.
- ingestion.ts: routing branch + pure sourceItemToConnectorKnowledgeItem
  mapper (exported for testing, same pattern as externalMeetingToRow).
- registry.ts: threads the new IngestionOutcome.knowledgeItems counter
  through syncContainer/syncNow's accumulation.
- Extended the existing ingestion.test.ts (fakeDeps, mocks, routing
  assertions) and added coverage for the new routing branch and mapper.

Couldn't run the vitest suite in this sandbox — it hits a pre-existing
rollup native-binary mismatch (linux-arm64 binary missing) unrelated to
this change, most likely because this sandbox bridges into a Linux VM
rather than your Mac's own arm64 darwin node_modules. Worth running
`npx vitest run` for these two files on your machine to confirm before
merging.

Still not built: the Supabase staging table and the real Vercel relay
tools (log_decision/log_risk/etc.) that will actually call
upsertConnectorKnowledgeItem via this routing — this commit only wires
the receiving end.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W3T66EZfvq3aX47UZKcxmM
… Brain

Registers a new zero-auth connector (littlebird-relay) that polls the
relay_staged_items table in Supabase (written to by the workbrain-relay
Vercel MCP server a Littlebird Routine calls) and routes staged
decisions/action items/risks/questions through the existing
upsertConnectorKnowledgeItem path (schema v55), exactly like any other
source. Delivery is at-least-once; ingestion is idempotent on
(connector_id, source_ref) so a redelivered row is a safe no-op.

Registered in initConnectors() with a 5-minute scheduled sync, mirroring
the HiNotes connector's pattern. Config is just a Supabase project URL +
its Secret key (required — the staging table has RLS enabled with no
policies, so only the Secret key can read it).

Validated with a standalone smoke test against a faked fetch (mapping,
cursor advancement, PATCH shape) since vitest cannot run in this sandbox
(pre-existing @rollup/rollup-linux-arm64-gnu platform mismatch unrelated
to this change) — please run `npx vitest run` on your Mac to confirm
before merging.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W3T66EZfvq3aX47UZKcxmM
Adds confidence vocabulary and provenance to the four first-class
knowledge tables (decisions/action_items/risks/questions) ahead of
wiring them to Empirical, per Work Brain architecture-discovery.md
section 6 item 2:

- Schema v56: `certainty` (confirmed/proposed/assumed/superseded,
  defaults to confirmed) and `evidence` (source quote/reference) on
  all four tables; `superseded_by` FK on decisions only, with a
  partial index.
- Migration 56 adds the columns idempotently via the existing
  getTableColumns guard for pre-v56 databases.
- KNOWLEDGE_CERTAINTY_VALUES + normalizeCertainty() validate/coerce
  incoming values (unknown input falls back to 'confirmed').
- ConnectorKnowledgeItem and packages/connectors' External* types
  carry the new optional fields end-to-end from connector source
  items through upsertConnectorKnowledgeItem's SQL.
- ingestion.ts passes certainty/evidence/supersededBy through
  sourceItemToConnectorKnowledgeItem for all four kinds.
- Tests added for certainty/evidence passthrough and null defaults.

Validated directly against SQLite (fresh-install schema, simulated
pre-v56 migration path, all INSERT/UPDATE variants) and via
`tsc --noEmit` on database.ts, since vitest cannot run in this
sandbox. Needs `npm run build` in packages/connectors and
`npx vitest run` on a real machine before merging.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W3T66EZfvq3aX47UZKcxmM
…st checks

- resolveClaudeCommand used POSIX path.join on Windows-style trusted-root
  paths; now uses win32.join on win32
- validateTargetDir couldn't canonicalize a protected path that doesn't
  exist yet, leaving an unresolved-symlink bypass in the ancestor check

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W3T66EZfvq3aX47UZKcxmM
…k suite

- reanalyze-exclusion.test.ts: mock brain-registry, not just brains helpers
  (stale after BrainRouter refactor)
- ingestion.test.ts: expected object missing schema v56 certainty/evidence
  fields
- handover-service.test.ts: assert against realpath, not raw mkdtemp path
  (macOS /var -> /private/var)
- download-service-session-c5.test.ts: existsSync mock made path-specific
  so queueDownloads' own disk-reconciliation logic isn't fooled into
  treating queued files as already synced
- meeting-recorder database.test.ts: full rewrite off a dead sql.js mock,
  now exercises the real better-sqlite3-backed DatabaseEngine directly

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W3T66EZfvq3aX47UZKcxmM
@coderabbitai

coderabbitai Bot commented Sep 5, 2026

Copy link
Copy Markdown

Important

Review skipped

Too many files!

This PR contains 108 files, which is 8 over the limit of 100.

To get a review, reduce the PR to 100 files or fewer by splitting it into smaller PRs or changing its base branch.

Upgrade to a paid plan to raise the limit.

This review couldn't start because sufficient usage credits or metered capacity aren't available. Add credits or update usage-based reviews in the billing tab, then retry.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Team

Run ID: a013cef5-4b88-4608-b48d-2804223c5c3d

📥 Commits

Reviewing files that changed from the base of the PR and between 8f657a0 and 7240334.

⛔ Files ignored due to path filters (1)
  • packages/hidock-mcp/package-lock.json is excluded by !**/package-lock.json
📒 Files selected for processing (108)
  • apps/electron/README.md
  • apps/electron/electron/main/ipc/__tests__/knowledge-graph-handlers.test.ts
  • apps/electron/electron/main/ipc/briefing-handlers.ts
  • apps/electron/electron/main/ipc/calendar-handlers.ts
  • apps/electron/electron/main/ipc/knowledge-graph-handlers.ts
  • apps/electron/electron/main/ipc/re-diarize-handlers.ts
  • apps/electron/electron/main/ipc/recording-handlers.ts
  • apps/electron/electron/main/services/__tests__/ai-provider-config.test.ts
  • apps/electron/electron/main/services/__tests__/artifact-service.test.ts
  • apps/electron/electron/main/services/__tests__/calendar-file-sync.test.ts
  • apps/electron/electron/main/services/__tests__/context-graph-service.test.ts
  • apps/electron/electron/main/services/__tests__/database.test.ts
  • apps/electron/electron/main/services/__tests__/download-service-session-c5.test.ts
  • apps/electron/electron/main/services/__tests__/gemini-models.test.ts
  • apps/electron/electron/main/services/__tests__/graph-provenance-cleanup.test.ts
  • apps/electron/electron/main/services/__tests__/graph-sync.test.ts
  • apps/electron/electron/main/services/__tests__/handover-service.test.ts
  • apps/electron/electron/main/services/__tests__/ingestion-atomic-current-marking.test.ts
  • apps/electron/electron/main/services/__tests__/ingestion-run-provenance.test.ts
  • apps/electron/electron/main/services/__tests__/ingestion-staleness.test.ts
  • apps/electron/electron/main/services/__tests__/knowledge-graph-service.test.ts
  • apps/electron/electron/main/services/__tests__/legacy-marker-migration.test.ts
  • apps/electron/electron/main/services/__tests__/promote-extraction.test.ts
  • apps/electron/electron/main/services/__tests__/reanalyze-exclusion.test.ts
  • apps/electron/electron/main/services/__tests__/reingestion-batch-transaction.test.ts
  • apps/electron/electron/main/services/__tests__/reingestion-discovery.test.ts
  • apps/electron/electron/main/services/__tests__/reingestion-scoped-deletion.test.ts
  • apps/electron/electron/main/services/__tests__/retry-backoff.test.ts
  • apps/electron/electron/main/services/__tests__/retry-classifier.test.ts
  • apps/electron/electron/main/services/__tests__/transcription-value-classification.test.ts
  • apps/electron/electron/main/services/__tests__/transcription.test.ts
  • apps/electron/electron/main/services/__tests__/whisper-cpp.test.ts
  • apps/electron/electron/main/services/actionable-eligibility.ts
  • apps/electron/electron/main/services/ai-provider-config.ts
  • apps/electron/electron/main/services/artifact-service.ts
  • apps/electron/electron/main/services/brains/__tests__/ollama-brain.test.ts
  • apps/electron/electron/main/services/brains/claude-code-brain.ts
  • apps/electron/electron/main/services/brains/ollama-brain.ts
  • apps/electron/electron/main/services/calendar-file-sync.ts
  • apps/electron/electron/main/services/config.ts
  • apps/electron/electron/main/services/connectors/__tests__/ingestion.test.ts
  • apps/electron/electron/main/services/connectors/hinotes/__tests__/hinotes-connector.test.ts
  • apps/electron/electron/main/services/connectors/hinotes/hinotes-connector.ts
  • apps/electron/electron/main/services/connectors/index.ts
  • apps/electron/electron/main/services/connectors/ingestion.ts
  • apps/electron/electron/main/services/connectors/littlebird-relay/__tests__/littlebird-relay-connector.test.ts
  • apps/electron/electron/main/services/connectors/littlebird-relay/littlebird-relay-connector.ts
  • apps/electron/electron/main/services/database.ts
  • apps/electron/electron/main/services/gemini-models.ts
  • apps/electron/electron/main/services/graph-sync.ts
  • apps/electron/electron/main/services/handover-service.ts
  • apps/electron/electron/main/services/knowledge-graph-service.ts
  • apps/electron/electron/main/services/ollama.ts
  • apps/electron/electron/main/services/recording-eligibility.ts
  • apps/electron/electron/main/services/retrieval-orchestrator.ts
  • apps/electron/electron/main/services/transcription.ts
  • apps/electron/electron/main/services/whisper-cpp.ts
  • apps/electron/electron/preload/index.ts
  • apps/electron/src/components/OperationController.tsx
  • apps/electron/src/components/layout/Layout.tsx
  • apps/electron/src/components/settings/ConnectorsSettings.tsx
  • apps/electron/src/pages/Settings.tsx
  • apps/electron/src/pages/Today.tsx
  • apps/electron/src/pages/__tests__/Settings.test.tsx
  • apps/electron/src/types/index.ts
  • apps/meeting-recorder/electron/main/__tests__/database.test.ts
  • docs/development/ADR_IPHONE_CALENDAR_BRIDGE.md
  • docs/development/ARCHITECTURE_DECISIONS.md
  • docs/development/IPHONE_CALENDAR_BRIDGE.md
  • docs/development/LOCAL_TRANSCRIPTION.md
  • eval/extraction-prompt-2026-09/COMPARISON.md
  • eval/extraction-prompt-2026-09/arm-a-llama3.2.ndjson
  • eval/extraction-prompt-2026-09/arm-c-gemma3-12b.ndjson
  • eval/extraction-prompt-2026-09/harness-run-a.mjs
  • eval/extraction-prompt-2026-09/harness-run-c.mjs
  • eval/extraction-prompt-2026-09/prompt-c.mjs
  • eval/extraction-prompt-2026-09/prompts-a-and-b.mjs
  • eval/extraction-prompt-2026-09/reingest-dryrun.mjs
  • eval/personal-content-eval/README.md
  • eval/personal-content-eval/RESULT.md
  • packages/ai-providers/src/complete.ts
  • packages/ai-providers/tests/complete.test.ts
  • packages/connectors/src/registry.ts
  • packages/connectors/src/types.ts
  • packages/database/src/eligibility.ts
  • packages/database/src/index.ts
  • packages/database/src/retrieval.ts
  • packages/database/tests/eligibility.test.ts
  • packages/hidock-mcp-legacy-tests/data-source.test.ts
  • packages/hidock-mcp-legacy-tests/mcp.test.ts
  • packages/hidock-mcp-legacy-tsup.config.ts
  • packages/hidock-mcp/README.md
  • packages/hidock-mcp/package.json
  • packages/hidock-mcp/src/database.ts
  • packages/hidock-mcp/src/eligibility.ts
  • packages/hidock-mcp/src/repository.test.ts
  • packages/hidock-mcp/src/repository.ts
  • packages/hidock-mcp/src/server.ts
  • packages/hidock-mcp/tsconfig.json
  • packages/knowledge-graph/src/extract.ts
  • packages/knowledge-graph/src/index.ts
  • packages/knowledge-graph/src/littlebird-adapter.ts
  • packages/knowledge-graph/src/littlebird-dedup.ts
  • packages/knowledge-graph/tests/extract.test.ts
  • packages/knowledge-graph/tests/littlebird-adapter.test.ts
  • packages/knowledge-graph/tests/littlebird-dedup.test.ts
  • packages/knowledge-graph/tests/personal-content-eval.test.ts
  • packages/knowledge-graph/tests/recording-provenance.test.ts

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@kelly1175 kelly1175 closed this Sep 5, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant