Adopt the shared SocratiCode index on co-index

[gregoryfoster]

Context

The cohort now has a shared SocratiCode index on a fifth VM, co-index
(index on the tailnet). It has been live and soaking since 2026-09-12 and
passed its 24h criterion today: 149 consecutive check-ins, zero non-202s, all
collections green, NRestarts=0 on every unit.

• Tracking issue: CannObserv/notifier#57
• Design doc, decisions D0–D14: docs/plans/2026-09-11-shared-qdrant-vm-design.md in notifier
• Reference implementation: notifier's .socraticode.json + .claude/settings.json

Adopting gets this repo two things it does not have today: a semantic index it
does not host (no Docker, no Ollama, no 9 GB image on your VM), and
cross-repo search — codebase_search with includeLinked: true answers a
question against every cohort repo in the store, not just this one.

This issue is a suggestion from the notifier repo. No commits have been made
here. Sequence and priority are yours.

---

1. Add .socraticode.json at the repo root

{
  "projectId": "watcher",
  "linkedProjects": [
    "../archiver",
    "../broker",
    "../replicator",
    "../notifier"
  ]
}

projectId is what names the Qdrant collections (codebase_watcher,
watcher_symgraph_file, …). Committing it is the point: every checkout
addresses the same collections regardless of where the working tree sits on
disk.

2. Add the six client variables to .claude/settings.json

{
  "env": {
    "QDRANT_MODE": "external",
    "QDRANT_URL": "https://index.taild0fb76.ts.net:6333",
    "OLLAMA_MODE": "external",
    "OLLAMA_URL": "http://index:11434",
    "EMBEDDING_MODEL": "nomic-embed-text",
    "EMBEDDING_DIMENSIONS": "768"
  }
}

These five are non-secret and should be committed — they are
self-documenting and travel with the checkout.

QDRANT_API_KEY goes in .claude/settings.local.json, which must be
git-ignored in this repo. Check it — do not assume:

git check-ignore -v .claude/settings.local.json   # must print a rule

This paragraph used to assert the file is git-ignored. In
CannObserv/broker it was not — no rule, no global core.excludesfile — and
that repo is public. Four of the five cohort repos carried the rule, which is
exactly what made the assertion read as true. Following this issue as
originally written would have put the cohort's single Qdrant key one
git add -A from GitHub, and per #57 a leak anywhere is a rotation
everywhere with no overlap window. Reported as
notifier#68; broker's own
half is CannObserv/broker#18.

If the rule is missing, add it before installing the key. Do not write the
key by hand — use notifier's
scripts/install_qdrant_key.sh,
which is the reviewed home for this step and now refuses any target git would
commit. See Prerequisites below.

3. Create sibling link stubs — not clones

Corrected 2026-09-14. This section originally said "clone the sibling
repos read-only". That is wasteful and slightly wrong; what follows is what
the tool actually needs.

includeLinked resolves each entry in linkedProjects to a path on disk and
uses it for exactly two things:

const linkedId = effectiveBaseProjectId(linkedPath);    // → collection name
collections.push({ name: collectionName(linkedId),
                   label: path.basename(linkedPath) }); // → display label

searchMultipleCollections then receives only {name, label} — no path
reaches it. Search content comes wholly from Qdrant. The directory is how the
tool learns which collection to query, nothing more.

So each sibling needs a directory containing one file:

../<sibling>/.socraticode.json    →  { "projectId": "<sibling>" }

No checkout, no fetch, no disk, nothing to keep in sync. Notifier's VM now
carries exactly this for archiver, replicator and watcher; copy the README.md
from there if you want the rationale sitting next to the stub.

Verified, not assumed. The real broker clone on notifier's VM was moved
aside and replaced with a directory holding a single 30-byte
.socraticode.json. codebase_search returned broker source with correct file
paths and line numbers, labeled [broker], with zero bytes of broker source on
disk.

A stub is also safer than a clone. A real checkout carries the sibling's own
.socraticode.json, so if that repo later changes its projectId, a stale
clone silently resolves to the old collection. A stub you wrote is a local
declaration and cannot drift behind a git pull nobody ran.

A stub naming a collection that does not exist yet is safe.
searchMultipleCollections catches per-collection failures and skips them, so
search keeps working while the other siblings adopt. The skip is logged to the
server's stderr and not surfaced in the tool result — so, as with trap 1, a
green result is not evidence every sibling answered.

4. Adopt the health hook

Copy .claude/hooks/socraticode-health.sh from notifier and wire it as a
SessionStart hook. It reports once per UTC day, never repairs, and exits 0 on
every condition so it cannot take a session down.

---

Traps

Every one of these was hit or measured during notifier's build. None is
speculative.

1. A missing linked directory is dropped silently. loadLinkedProjects in
socraticode/dist/config.js filters on fs.existsSync with no warning. Until
2026-09-14 notifier's .socraticode.json listed four siblings while only
../broker existed on its VM, so includeLinked: true had been quietly
searching one repo, not four, and reporting nothing amiss. Stubs for the other
three are now in place there. Verify your own stub set rather than trusting a
green result.

2. Adopting projectId renames your collections. Resolution order is
SOCRATICODE_PROJECT_ID env > .socraticode.json > SHA-256 of the absolute
path. If this repo was ever indexed without the config file, it is sitting in
path-hash collections that the rename orphans. Run codebase_remove,
codebase_graph_remove and codebase_context_remove before adding the
file, then re-index — or the old set lingers in the shared store forever, with
a name nothing maps back to a repo.

3. QDRANT_URL, never QDRANT_HOST. QDRANT_MODE=external refuses to
start without a URL, and the fallback it would otherwise build is
`${QDRANT_API_KEY ? "https" : "http"}://${QDRANT_HOST}:${QDRANT_PORT}`
with QDRANT_PORT defaulting to 16333, not 6333. An API key with no URL
therefore assumes https against the wrong port, and the error reads like a
network fault.

4. Use the full MagicDNS name. https://index.taild0fb76.ts.net:6333, not
https://index:6333 — the short name is not in the certificate's SAN. Qdrant
serves TLS because SocratiCode refuses to send QDRANT_API_KEY over a
non-TLS, non-localhost connection (D14; this killed notifier's first index
attempt 20 s in).

5. Never set QDRANT_COLLECTION_PREFIX or SOCRATICODE_BRANCH_AWARE=true.
The prefix is prepended to the global socraticode_metadata collection too,
so one VM setting it splits the cohort namespace silently. Branch-awareness
appends the branch name to the project id — a fresh collection set per branch.
Both are failure modes every health check still calls green. notifier pins this
with a test (tests/deploy/test_socraticode_config.py); consider the same.

6. The env block applies only in a trusted folder. On a freshly cloned
VM, an untrusted workspace means the block does not apply, QDRANT_MODE
reverts to managed and OLLAMA_MODE to auto — and SocratiCode then tries
to start Docker containers on a host that has none, rather than reporting
missing configuration. Trust the folder, then confirm codebase_health names
the external endpoints and not a container.

7. The health hook skips a configured project with no toolchain — silently.
gregoryfoster/skills#281:
9 days, 5 sessions, 0 session lines. Until that is fixed, a green session start
is not evidence the hook ran.

8. Ollama on index has no authentication. Port 11434 is reachable by any
node the ACL admits. It is an embedding endpoint on a private tailnet, which is
the accepted posture — but it is not authenticated, so do not treat tailnet
reachability as authorization.

---

Prerequisites — operator, tracked in notifier#57

These are not yours to do; listed so a failure is recognizable rather than
mysterious.

• Tailnet ACL must admit this repo's VM to tag:index:6333,11434. A peer is
  visible only through an acls rule, and a missing one presents as a DNS
  failure, not a permission denial (archiver#193, replicator#88).

• QDRANT_API_KEY delivered to this VM out of band. Qdrant holds a single
  global service.api_key — no key list, no per-client identity, no
  per-collection scope — so every cohort VM holds the same secret and a leak
  anywhere is a rotation everywhere. Rotation has no overlap window: every
  client 401s from the restart until it is updated.

  Use notifier's scripts/install_qdrant_key.sh rather than writing the value
  by hand. Run from an operator machine, the only host able to reach both ends
  — exe.dev VMs are isolated from each other, and Visual diff: screenshot comparison between snapshots #57 D13's tag:index:22 edge
  was retired after the soak:

  scp scripts/install_qdrant_key.sh <vm>.exe.xyz:~/
  ssh co-index.exe.xyz \
      "sudo sed -n 's/^QDRANT__SERVICE__API_KEY=//p' /etc/socraticode/qdrant.env" \
    | ssh <vm>.exe.xyz 'bash ~/install_qdrant_key.sh ~/watcher'

  The key travels on stdin, never argv — an argument is visible in ps to
  every other process for the life of the call and lands in the caller's shell
  history. The script merges into the existing JSON atomically at 0600, so a
  crash cannot leave a truncated key.

  Expect installed 64 chars. Any other length is a truncated transfer,
  which 401s exactly like a wrong key; the length is the only cheap
  discriminator between the two. And never run it under bash -x — tracing a
  script that touches a credential writes the value to stdout, which is how two
  were leaked during Visual diff: screenshot comparison between snapshots #57.

  Full walkthrough including rotation: notifier's
  docs/COMMANDS.md.

---

Acceptance

• git check-ignore -v .claude/settings.local.json prints a rule
• install_qdrant_key.sh reported 64 chars
• codebase_health reports the external Qdrant and Ollama, no containers
• codebase_index completes; collections named codebase_watcher etc. appear in the shared store
• any pre-existing path-hash collections for this repo are removed
• codebase_search answers a question about this repo
• a link stub exists for every entry in linkedProjects
• codebase_search with includeLinked: true returns hits from at least one sibling
• the health hook prints on a fresh session (or is confirmed skipped per trap 7)

[gregoryfoster]

Update from upstream: gregoryfoster/skills#281 shipped in gregoryfoster/skills@219c6c5. Three parts of this issue change once skills-vendor/gregoryfoster-skills passes 219c6c5. The pointer is at b4f5cf0 today, 39 commits behind upstream main.

• §4 says to copy .claude/hooks/socraticode-health.sh from notifier. Don't copy it. This repo already carries the hook as a symlink into the vendored skill (→ ../../skills-vendor/gregoryfoster-skills/skills/init-socraticode/scripts/socraticode-health.sh, mode 120000), registered as a SessionStart hook in .claude/settings.json. notifier's is the same symlink, so copying would swap a link that updates on every refresh for a frozen file. That file would never receive the socraticode-health.sh: a configured project with no toolchain is skipped silently (9 days, 5 sessions, 0 session lines) gregoryfoster/skills#281 fix, or any later one. Here, §4 comes down to the submodule bump.
• Trap 7 is fixed. Past the manifest gate, a host with no node now hears this once a UTC day: node is not on PATH, but this project is configured for SocratiCode … the codebase_* tools will fail — answer code questions with grep/rg this session, along with a pointer to preflight.sh --check. A missing mcp-driver.mjs says nothing was measured. So after the bump, "confirmed skipped per trap 7" no longer applies to the acceptance item: a fresh session either prints the node line or runs the check.
• Trap 1 is now reported. The daily check reads .socraticode.json's linkedProjects and SOCRATICODE_LINKED_PROJECTS the way loadLinkedProjects() does. It names every entry that does not resolve on the host: linkedProjects — N of M resolved (missing: <path> (<source>), …). It runs only when node and the driver are present, and it cannot tell whether a link that does resolve was ever indexed. So "verify your clone set" still stands for that half.

[gregoryfoster]

Status here, as of 2026-09-16: not switched over. There is no .socraticode.json on main, no client variables in .claude/settings.json, and no watcher collections on co-index. With #296 done, the first launch would be on co-watcher, so items 1-4 apply to that VM: check its memory and your unit's OOM score there first.

What broker found by adopting first

Broker (CannObserv/broker#17) finished its adoption on 2026-09-16. codebase_broker (476 points), context_broker (137) and the three broker_symgraph_* collections are on co-index, all green. It went badly before it went well, and the first item below applies to any cohort VM that runs a production service beside the dev sessions that launch SocratiCode.

1. Launching the server uncapped took a production VM down

On broker's then-2 GB, no-swap VM, every SocratiCode server launch pushed it over the edge: the plugin's first start, preflight's claude mcp list, and mcp-driver.mjs index.

• Nothing was OOM-killed. The kernel failed atomic allocations in tailscaled and ksoftirqd, so the network path degraded while every process stayed alive.
• Outage: the bus was effectively gone for 57 m 48 s, and replicator did not reconnect on its own (Replicator did not reconnect after the broker's 2026-09-16 outage and restart - restart it, then validate reconnection as a property replicator#94, since fixed).
• Aftermath: broker went to 8 GB and added memory protection (Protect the broker from dev tooling: memory headroom, an atomic reserve, and a killer that prefers node over redis broker#21 and Add savepoint boundary for notification dispatch in check pipeline #25).

Before your first launch, check the VM you'll launch on: free -m (and whether there is swap), what production service shares it, and the next item.

2. On exe.dev, dev tooling is at oom_score_adj -1000, and your service is not

Every process started from an exe.dev session inherits -1000 from exe-init and sshd: VSCode Server, Claude Code, and the SocratiCode server they start.

• The kernel's OOM killer cannot pick any of them. Under real memory exhaustion it kills your service instead.
• A cgroup memory cap on such a process stalls it rather than killing it.
• earlyoom 1.7 doesn't fix this on its own. It floors a --prefer match at 300, while a service at the default adj 0 reads ~667 on this kernel, so it too would kill the service first. earlyoom --dryrun -d on broker showed exactly that until Add savepoint boundary for notification dispatch in check pipeline #25.

cat /proc/self/oom_score_adj                                              # from a session shell: -1000
cat /proc/$(systemctl show -p MainPID --value <your-service>)/oom_score   # ~667 at adj 0

Broker's fix is OOMScoreAdjust=-900 on its production units, with the reasoning in deploy/README.md Memory protection.

3. What a capped run costs, measured

Broker's successful index ran the cold npx install, the server and mcp-driver.mjs index together in a transient user service:

• Limits: MemoryHigh=1200M, MemoryMax=1536M, CPUQuota=100%, OOMScoreAdjust=500. The adj override matters; see item 2.
• Cost: 7 min wall and 13.9 s CPU, since embedding runs on co-index. Process memory peaked at ~610 MB, plus ~519 MB of page cache from the npm install.
• Where the pressure came from: every memory-throttle event came during the npm install, none during indexing.
• Bus impact: none. The probe found nothing, every PING came back in ≤6 ms, and all consumer groups stayed caught up.
• Scaling: broker is the smallest cohort repo (25 graph files), so expect the index phase to grow with yours.

The full invocation is in the #17 comment. systemd-run --user --scope -p MemoryMax=… choom -n 500 -- <cmd> also works from a session, keeping stdio and environment as they are. Both need linger (loginctl show-user exedev -p Linger) and memory/cpu in /sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/cgroup.controllers.

4. The everyday launch paths are still uncapped, and there is no settled design yet

Once adopted, the server launches:

• from the plugin, once per Claude Code session;
• from the daily socraticode-health.sh hook;
• from preflight's claude mcp list.

All of these are uncapped. The plugin also runs npx -y --prefer-online socraticode@latest, so a launch can mean a fresh install of a new version: the plugin cache here is 1.13.3, and the launch pulled 1.14.0.

CannObserv/broker#27 designs a capped, pinned launch for the cohort. Until it lands, broker keeps the plugin disabled and disableAllHooks set locally, and runs index jobs capped by hand. On a small or production-sharing VM, consider doing the same rather than enabling the plugin as §4 of this issue describes.

5. Smaller things

• Bump vendored skills to d04cebf or later (gregoryfoster/skills main).
  • It carries skills#287: preflight in external-store mode no longer demands Docker, and projectId and linked projects are handled.
  • It carries skills#293: the auto-refresh hook now pushes the pointer bump it commits. That unpushed commit is what made replicator's check_main_checkout.sh guard refuse restarts in replicator#94.
• If Docker is installed but unused, check systemctl is-enabled docker.socket. While the socket is enabled, any docker CLI call, even docker ps, starts dockerd and containerd (~120 MB). Broker disabled the socket; skills#287's preflight asks systemd instead of probing.
• codebase_broker now exists, so a ../broker link stub resolves to real results instead of being skipped silently.