feat: mine every profile's reasoning, pick injection targets, stop leaking embeddings - #156
Merged
Merged
Conversation
…aking embeddings Reasoning mining only ever scanned ~/.claude/projects, so a machine running a second Claude Code profile (the normal way to use a different vendor's models) had every trace from that profile invisible — no setting made it visible. New [reasoning_training] extra_transcript_dirs lists additional profile roots, each being the directory that holds projects/. Roots are de-duplicated, missing ones are skipped rather than fatal, and each file's project attribution resolves against its own root — flattening per-root discovery without that would mis-label one profile's traces as the other's. The injection map's target was a free-text box while the source was a dropdown. It is now a combobox: it offers every detected model but stays typable, because targets are matched with fnmatch — a glob and a model not yet seen in any transcript both have to remain enterable. Targets are deliberately not filtered by has_thinking_text; that gate is for sources, and a target is the recipient. Separately, embeddings silently inherited an ambient OPENAI_BASE_URL. That env var is how people point some OTHER tool at a proxy, so with no surreal-memory endpoint configured a brain's memory text went to a host the operator never chose, failing later as a confusing error from a vendor that had never heard of the configured model. OpenAIEmbedding now always passes an explicit base_url and warns when it ignores OPENAI_BASE_URL. The Stop hook's loopback privacy gate was defeated by the same omission — it validated an endpoint then built the provider without passing it, so the constructor re-resolved from the environment alone; it now hands over the checked endpoint. The dashboard's connection test had the same bug with no gate at all. smem reindex also hid why it failed: one identical low-information line per batch, the real exception swallowed, thousands of neurons ground through repeating it, and exit 0 regardless. It now reports the real error once, aborts after a few consecutive failures when nothing has embedded, and exits non-zero on a total wash-out. Bumps version to 3.2.0. BREAKING for one setup: relying on OPENAI_BASE_URL alone to reach a LiteLLM/vLLM/Azure server now routes to api.openai.com instead — set [embedding] endpoint or SURREAL_MEMORY_EMBEDDING_ENDPOINT to restore it.
…as an unused arg The multi-root fixture was named _home, whose leading underscore signals side-effect-only — but most tests genuinely use its value, so ruff's PT019 fired on every one of them. Renamed to home_root for the tests that use it, and the single side-effect-only test now declares it via usefixtures. Caught by CI because the lint step covers src/ AND tests/; the local check that missed it had only been run against src/.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three related fixes, all found while investigating a report that reasoning mining "only offers Anthropic models".
1. Mining now sees every Claude Code profile
reasoning_minerresolved exactly one root —~/.claude— so a machine running a second profile (the normal way to use a different vendor's models on one box) had every trace from that profile invisible, and nomining_modelssetting could reveal it.New
[reasoning_training] extra_transcript_dirs(alsoSURREAL_MEMORY_REASONING_EXTRA_DIRS) lists additional profile roots — each entry is the directory that holdsprojects/, e.g.~/.claude-ZAI.Two traps handled deliberately:
_project_from_pathresolves against aprojects_dir. Flattening per-root discovery without carrying the owning root would silently mis-label one profile's traces as the other's. The project name is now resolved at discovery time, while the root is still known — with a regression test that puts a same-named project dir in both roots.~/.claudeagain as an extra can't scan it twice. Scan-state keys are absolute paths, so two roots can't collide there.Verified against a real second profile: 150
glm-5.2traces became visible, alongside the existing Anthropic ones.2. Injection target is a combobox, not a text box
The source was a dropdown; the target was a bare
<input type="text">.It's now a combobox (
<datalist>) offering everydetected_modelsentry while remaining free-text, because:fnmatch, so a glob (glm-*) is a legitimate value a plain<select>would have taken away;Targets are deliberately not filtered by
has_thinking_text— that gate applies to sources; a target is the recipient and may emit no thinking of its own.3. Embeddings stopped inheriting a stranger's endpoint
With no surreal-memory endpoint configured,
OpenAIEmbeddingomittedbase_urlentirely, so the OpenAI SDK fell back to the ambientOPENAI_BASE_URL— the ecosystem-standard knob for pointing some other tool at a proxy. Result: a brain's memory text sent to a host the operator never configured for embeddings, surfacing only as a confusing error from a vendor that had never heard of the configured model.OpenAIEmbeddingnow always passes an explicitbase_url(defaulthttps://api.openai.com/v1) and warns when it ignores a setOPENAI_BASE_URL.config.tomltherefore cleared the gate while the client pointed elsewhere. It now hands over the checked endpoint.4.
smem reindexstopped hiding why it failedIt printed one identical
batch N-M failed (skipped)line per batch with the actual exception swallowed, ground through every remaining neuron repeating the same failure, and exited 0 regardless. This cost two full diagnostic rounds in practice. It now reports the real error once, aborts after a few consecutive failures when nothing has embedded, and exits non-zero on a total wash-out.Deliberately rejected
A reviewer pass suggested also changing
_create_providerto passbase_url=endpointinstead ofendpoint or None. Verified as a no-op ("" or env≡None or env) and dropped rather than adding churn.Breaking change (one setup)
If you relied on
OPENAI_BASE_URLalone to point surreal-memory at a LiteLLM/vLLM/Azure/self-hosted OpenAI-compatible server —provider = "openai"with no endpoint in eitherconfig.tomlorSURREAL_MEMORY_EMBEDDING_ENDPOINT— embeddings now go toapi.openai.cominstead. Set[embedding] endpoint(or the env var) to restore the old routing; that's the same one-key change the new warning names. This is why the bump is MINOR, not PATCH.Test plan
check_dead_modules.py, security scan — all cleanmkdocs build --strictclean; all four doc generators re-runtsc -bandeslintclean; bundle rebuilt (it is committed)_ensure_clientto the old shape and confirmed 2 of the 5 new isolation tests fail, then pass again after restoring — a test that cannot fail proves nothingclaude_diroverride still suppressing extras, missing root skipped, duplicate root not double-counted, async ingest path