memU embeddings: route through MEMU_EMBEDDING_BASE_URL and backfill NULL rows

[alex-fedotyev]

Summary

memU stopped writing embeddings on 2026-05-06. Every memorize call from then on persisted embedding_json=NULL, vector search at recall time was disabled, and memory_recall returned "No relevant memories found" for any query that should have hit a memory written in the last few days.

The 2026-05-05 sidecar work (Ollama at http://embeddings:11434/v1) actually shipped on the host docker-compose.yml: the env var MEMU_EMBEDDING_BASE_URL was set on the agent container, the Ollama service was running and healthy, and the model returned valid 768-dim vectors. But the in-process nerve.memory.memu_bridge code never read the env var. It gated the embedding LLM profile, the "rag" retrieve method, and the categorize_items step exclusively on self.config.openai_api_key. With OpenAI cleared (the sidecar was supposed to replace it) the bridge took the no-embed branch and persisted every new item with embedding_json=NULL. RCA in notes/lessons/2026-05-09-memu-embeddings-not-wired.md.

5,100 of 6,888 memu_memory_items rows on the running container have NULL embeddings; the 879 with embeddings all date to 2026-05-05 when the OpenAI key was still active. Live memory_recall("AI Summary HDX-3992 redactSecrets", 5) returned no results despite a known matching memory.

What this PR does

Routing fix (commit fbb74d7): resolve the embedding endpoint as env -> YAML -> OpenAI in one place, set a single embeddings_configured flag, and key every downstream behavior off that flag.

• nerve/config.py: MemoryConfig gains embedding_base_url, embedding_api_key, and llm_concurrency fields. Env vars MEMU_EMBEDDING_BASE_URL, MEMU_EMBEDDING_API_KEY, MEMU_EMBED_MODEL override the YAML defaults at runtime so the docker compose path doesn't have to rewrite config.yaml on every container restart.
• nerve/memory/memu_bridge.py: _initialize reads the env vars first, registers the embedding profile against the resolved endpoint, and falls back to OpenAI only when neither env nor YAML is set. _has_embeddings, _categorize_no_embed, the retrieve config method ("rag" vs "llm"), and memory_extract_llm_profile all key off the same unified flag.
• A bounded asyncio.Semaphore wraps memU's chat calls so the per-memory-type fan-out (a 4-way gather in memU's extract_items pipeline) can't blow the Anthropic rate limit on lower API tiers. Configurable via memory.llm_concurrency, default 1. Re-instrumentation reuses the same Semaphore so callers already queued don't lose their slot. SDK max_retries bumped from 0 back to 4 (with concurrency bounded, retries actually drain the queue instead of stacking).
• nerve/bootstrap.py::_build_docker_compose now writes the embeddings service block, the MEMU_EMBEDDING_* env vars on the nerve service, depends_on: embeddings: condition: service_healthy, the ~/.nerve/claude mount for persisted Claude Code state, the path-aligned ${HOME}/nerve-workspace and ${HOME}/projects mounts, and the /var/run/docker.sock mount. The entrypoint creates /root/* symlinks pointing at HOST_HOME so hardcoded /root/nerve-workspace and /root/projects paths still resolve. Brings nerve init regeneration in line with what the live host docker-compose.yml already has.

Backfill (commit db3f588): scripts/backfill_memu_embeddings.py walks memu_memory_items (text source: summary) and memu_resources (text source: caption) for rows with NULL or empty embedding_json, batches 32 at a time, posts to {MEMU_EMBEDDING_BASE_URL}/embeddings with the OpenAI-compatible payload Ollama accepts, and writes the resulting vectors back. Idempotent; supports --dry-run, --limit, --table. Single transaction per batch so an interrupt loses at most one batch.

Test plan

• Unit: pytest tests/test_memu_bridge.py tests/test_bootstrap.py -> 115 passed, 0 failed. Full suite: 447 passed, 2 skipped (the only failure was a pre-existing test_cli_upgrade.py::test_docker_mode_bails_out unrelated to these changes).
• Backfill dry-run on the running container's memu.sqlite: reports 6,009 memu_memory_items + 329 memu_resources pending (excluding 103 resources with NULL captions). Counts match the SQL prior to my run.
• Backfill --limit 100 --table memu_memory_items: wrote 100 768-dim vectors in 1.5s. Sample row spot-check: vector length 768, leading dims [0.0096, 0.037, -0.132]. Re-running --dry-run reports 5,909 pending, confirming idempotency (5,909 = 6,009 - 100).
• Endpoint sanity: curl POST http://embeddings:11434/v1/embeddings with batch input returns 768-dim vectors per input, response shape matches OpenAI's contract.
• Live memorize round-trip on the running container: deferred. The fix lives in this branch and the running daemon is on claude/engine-sdk-resume-guard with the buggy bridge. Restarting the agent kills the active chat session per TASK.md, so I'll let the user pick the moment to recreate the container with the new code. Once restarted, a new memorize call should write a non-NULL embedding_json of length 768 and memory_recall against that memory should return it.
• Full backfill (~6,338 rows): worth running once after the user restarts the container so historical recall works. The script's progress logging makes it easy to resume.

Notes for the reviewer

• The original 2026-05-05 stash also bundled a nerve-services = nerve.services:main console-script entry in pyproject.toml and several comments referencing a docker-mcp sidecar. nerve/services.py lives only on the abandoned alex/docker-mcp-spike branch, so installing with that entry would break pip install -e. Dropped here, with the dangling sidecar comments rewritten to describe the actual current setup (direct host socket mount).
• The engine-SDK-resume-guard half of the original stash already shipped on f39e62b and is excluded.
• Tier estimate: ~250 prod lines added. Single concern (memU embedding routing). The bootstrap.py delta is the largest part and is mostly compose YAML in an f-string plus an entrypoint shell snippet, not new branching logic.
• After merge, I'll update notes/outcomes/2026-05-workflow-automation-loop-closure.md AC14 to PASS once the user restarts the container and confirms a recall round-trip works.

Closes the recall regression tracked under task memU: replace OpenAI embeddings with a self-hosted free embedding sidecar.