fix(query): score the rationale attribute, not just labels (#2293) - #2294
Open
nuboxworld-byte wants to merge 1300 commits into
Open
fix(query): score the rationale attribute, not just labels (#2293)#2294nuboxworld-byte wants to merge 1300 commits into
rationale attribute, not just labels (#2293)#2294nuboxworld-byte wants to merge 1300 commits into
Conversation
…raphify-Labs#1753) build_from_json's ghost-node merge iterated set(G.nodes()), so when two nodes shared a (basename, label) key the "canonical" survivor was chosen by CPython's per-process string-hash order — rebuilding the same extraction JSON in a fresh process could pick a different survivor, silently changing which node id represents a concept. That breaks any workflow persisting ids across a rebuild; concretely it broke the cluster->relabel step (community membership referenced an id that the second build merged away -> KeyError in report generation). Two changes: - Pass 1 and Pass 2 now iterate sorted(node_set), not set(node_set), the same deterministic-order fix the edge loop below already uses on purpose. - The Graphify-Labs#1257 ambiguity guard is extended to the case it did not cover: two NON-AST nodes sharing a key but from DIFFERENT source files are distinct concepts, not an AST ghost/canonical twin, so the key is marked ambiguous and both survive rather than one arbitrarily merging away (data loss). A genuine same-file duplicate (identical source_file) is not flagged and still collapses to one node. Reported with a precise root-cause, minimal repro, and real-world impact by @erasmust-dotcom. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…dep (Graphify-Labs#1745) When the [sql] extra is absent, .sql files are counted as code and scanned but extract_sql returns an error result and zero nodes — and the graph builds "successfully" with the entire SQL corpus missing. Neither existing warning catches it: Graphify-Labs#1666's zero-node warning skips results carrying an "error", and Graphify-Labs#1689 only covers files with NO extractor at all (.sql HAS a dispatch entry). extract() now scans per-file results for a "not installed" error, groups the affected files by extension, and prints a warning naming the extra that restores the language (pip install "graphifyy[sql]"), via a small _EXTRA_FOR_EXTENSION map (sql, terraform, dm). The map is only consulted after an extractor actually reports the dependency missing, so it can't mislabel a language that has a working fallback. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…raphify-Labs#1749) The extraction spec forbids cross-language `calls` edges, and build already dropped cross-language INFERRED `calls`. But `imports`/`references` had no such guard: an unresolved Python `import time` resolved by bare stem (the Graphify-Labs#1504 old-stem alias) onto a `src/time.ts` file node, welding a polyglot repo's two language halves together. In the reporter's repo three such edges were the only bridge between 2409 Python and 1403 TS nodes, so every backend<->frontend shortest path routed through time.ts, inflating its betweenness ~90x and making it the Graphify-Labs#1 reported god node. Hoist the interop-family map to a module constant and extend the edge-loop guard to `imports`/`imports_from`/`references`. For these relations the edge is dropped only when BOTH endpoints are known code languages of different families, so a config/manifest -> code reference (unknown ext) is never mistaken for a phantom. `calls` behavior is unchanged (still INFERRED-only, still drops when either family differs). Regression tests: py->ts import dropped, ts->ts import kept, config->code reference kept. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…CWD (Graphify-Labs#1747) Case 1 — `extract <corpus> --out <dir>`: the graph went to <dir> (cache_root is already passed to the AST extractor), but detect()'s word-count/stat-index cache uses the scan root, so a stray graphify-out/cache/ was created inside the corpus (and left behind even when the run aborted at the no-LLM-key gate). Thread an optional cache_root through detect() -> cached_word_count() -> _ensure_stat_index() and pass out_root from the extract CLI, so the stat index lives under --out. Entry keys are absolute paths, so relocating the index file is safe. Case 2 — `cluster-only --graph <elsewhere>/graphify-out/graph.json`: outputs (GRAPH_REPORT.md, re-clustered graph.json, labels, analysis, html) were written to the CWD's graphify-out/, ignoring where --graph lives. They now write beside the input graph when it sits in a graphify-out/ dir (another project/tenant's output), while still falling back to the CWD for an arbitrary archived backup/graph.json — the restore-into-place workflow Graphify-Labs#934 pins. Regression tests for both cases; Graphify-Labs#934 still passes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…bs#1696) Java member calls (`gw.charge()`) resolved by bare method name, so a call bound to any same-named method in the corpus — e.g. `PaymentGateway.charge` and `AuditLog.charge` were indistinguishable, producing phantom cross-class edges and a false god node. The extractor now preserves the receiver and its static type, and _resolve_java_member_calls binds the call against the receiver's declared type: explicit-type receivers and `this` are exact; current-class fields, method parameters, and explicitly-typed locals resolve via a method-scoped type table; a missing/ambiguous/inherited/chained receiver is skipped rather than falling back to a bare name match (same single-owner god-node guard as the C#/Swift/Ruby resolvers). Fully-qualified and nested-type receivers are deferred since they need package- and nesting-aware type identity. Verified: `gw.ping()/gw.charge()` (gw: PaymentGateway) bind to PaymentGateway, the three charge() calls dedup to one edge, and no edge targets the same-named AuditLog methods. Applies cleanly to the post-Graphify-Labs#1737 layout (extract.py + extractors/engine.py). 13 new tests; full suite 3135 passed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ify-Labs#1755) `graphify update` (the watch._rebuild_code / _reconcile_existing_graph path) evicted every hyperedge whose source_file is in the corpus, because on a full update every corpus file counts as "rebuilt" and hyperedge eviction reused the node/edge eviction set. But the AST pass never emits hyperedges, so nothing replaced them — doc-sourced hyperedges (what semantic extraction produces) were permanently lost on the first update after a full build, even on a no-op run. Split out a hyperedge_evicted_source_identities set scoped to genuinely deleted (and symlink-target-outside) sources only, not merely-rebuilt ones. Replacement- by-id (new_hyperedge_ids) and dangling-member cleanup are unchanged, so a real semantic re-extraction still replaces its own hyperedges and orphaned ones are still dropped. Parametrized regression test (full + incremental doc update). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…s#1764) extract_json emitted `imports` edges for package.json dependencies and `extends`/`$ref` edges for tsconfig.json to target ids (`_make_id("ref", ...)` / `_make_id(key)`) that it never created as nodes. build_from_json drops edges to unknown node ids silently (that case is filtered out of real_errors), so dependency and extends structure vanished from the graph on two of the most common files in any JS/TS repo, surfaced only by diagnose_extraction after the fact. The extractor now adds the referenced target as a `concept` node (external ref, not a corpus file) before emitting each edge, so the edges survive build. Regression test asserts no dangling endpoints, the concept nodes exist, and the import/extends edges land on real targets with no self-loops. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…t references edges (Graphify-Labs#1746) The live-introspection FK query joined information_schema.referential_ constraints, which Postgres only exposes for constraints where the current user has WRITE access to the referencing table. A read-only introspection role therefore got zero FK rows — while tables/views/routines still appeared (SELECT is enough for those views) — so the graph silently lost every `references` edge, contradicting the documented FK-mapping behavior. Switch to pg_catalog.pg_constraint (world-readable, not privilege-filtered), keyed by constraint oid rather than name — which also fixes a latent bug where same-named constraints on sibling tables could cross-match in the old name-based key_column_usage joins. Composite-FK column order is preserved with UNNEST(conkey/confkey) WITH ORDINALITY. Mock test asserts the query targets pg_constraint and not the privilege-filtered view, plus composite-FK ordering. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The badge's href pointed at www.ycombinator.com/companies/graphify, which 404s — the public S26 company page isn't published yet. Show the badge without a link rather than ship a dead click; re-add the href once YC publishes the page. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…aphify-Labs#1775) The TypeError reported on 0.1.14 is already fixed on v8: sanitize_label coerces None ('if text is None: return ""') and the source_file call site guards with str(data.get("source_file") or ""). Add regression tests (unit + to_html integration with null label/source_file) so it can't silently regress. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tion (Graphify-Labs#1781) _rewire_unique_stub_nodes gated merge targets through _is_type_like_definition, which rejects any label ending in `)`. So a function referenced from another module (passed by name, e.g. FastAPI's Depends(get_db)) left its reference edge dangling on a sourceless name-only stub while the real def had zero incoming edges — "who references this function" returned nothing. Class/type symbols were fine; only functions/methods suffered. Top-level function defs (label `name()`, not `.name()` methods or `Class.m()` qualifiers) are now eligible rewire targets, but only when: - the label key matches exactly one such function corpus-wide (existing unique-candidate guard — two same-named functions stay unresolved), AND - the candidate shares a language family with the stub's referrers, so a Python `get_db` reference can't bind to a unique Go `get_db()` (Graphify-Labs#1718/Graphify-Labs#1749 interop guard), AND - the stub is not used as a supertype (inherits/implements/extends) — you don't inherit from a function. Types are unchanged. Regression tests: cross-module function ref binds to def; cross-language, ambiguous, and supertype cases all correctly left unresolved. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The reported bug — a method chained directly onto a `new X(...)` expression (no intermediate variable) producing no calls edge — is already fixed on v8: `new Merger(ctx).Combine(...)` emits calls -> Merger.Combine. Add a regression test so the fluent new-expression receiver stays covered. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…raphify-Labs#1784) .rake files are plain Ruby (Rake's task DSL is ordinary method calls), but the extension was gated out everywhere, so rake tasks were classified as unsupported, skipped, and their calls invisible. Add `.rake` to all seven `.rb` gates the reporter mapped: - detect.CODE_EXTENSIONS (classification) - extract._DISPATCH (extractor dispatch) - extract._LANG_FAMILY_BY_EXT-adjacent language-name map (.rake -> ruby) - the ruby_member_calls LanguageResolver suffix set - both `.rb`-suffix filters in ruby_resolution.py (raw-call gather + class-def index) - analyze language-stats map - build repo-tag map The extractor already parsed the content; this is purely extension routing. Regression test: a `.rake` task's `Widget.tally` resolves cross-file to the `.rb` definition. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
extract_bash only created a cross-file edge for `source x.sh` / `. x.sh`. The two most common ways one script runs another — `bash x.sh` and `./x.sh` — produced no edge, so in any repo where scripts invoke each other by execution the call topology was missing (each script left an isolated file+entry pair). Emit a `calls` edge (context `script_invocation`) from the caller's entry (or enclosing function) to the invoked script's entry node, for script-runner commands (bash/sh/zsh/ksh/dash <path>) and bare `./x.sh`, but only when the target resolves to a real .sh file on disk — so no phantom edges to missing or function-shadowed names. Verified end-to-end: the edges land on real target nodes (no dangling drop at build). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nt (Graphify-Labs#1768) suggest_questions()'s "isolated/weakly-connected nodes" filter was missing the `file_type != "rationale"` exclusion that report.py's Knowledge Gaps section already applies, so the same GRAPH_REPORT.md reported two different counts for the same concept (757 vs 245 on a real graph) — an internal inconsistency that made a healthy graph look like a documentation problem. Add the same filter so both computations agree. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nts (Graphify-Labs#1785) `graphify path` committed each endpoint to _score_nodes()[0]. The full-query bonus tier only fires when the query equals/prefixes a label, so a query that is a token subset of the intended label ("Reject-everything judge" vs "Degenerate Reject-Everything Judge") got no bonus and a node prefix-matching one rare token ("Rejection Summary") could out-score it on IDF alone — anchoring the path on an unrelated, often disconnected node and returning a false "No path found". _pick_scored_endpoint() scans the score-ordered list and takes the first candidate whose label contains EVERY query token, falling back to scored[0] when none does — so when the head already full-matches (the common case) resolution is unchanged. Wired into both the `path` CLI and the MCP _tool_shortest_path. The close-runner-up ambiguity warning now fires only when the picked endpoint is the raw score head (a full-token override was chosen on coverage, not score, so the head's margin is irrelevant). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…aphify-Labs#1789) The absolute-path-in-node-ids leak reported on 0.8.19 is already fixed on v8: detect() returns paths relative to the scan root, so the CLI-produced graph.json uses relative structural node ids (portable, no username/home leak). Lock it with a regression test that extracts the same corpus from two different absolute checkout dirs and asserts identical, leak-free node ids. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…y-Labs#1796) build_merge already drops a re-extracted file's stale base nodes before merging (replace-per-source), so on current code an EDITED file passed only in new_chunks is handled correctly. But the prune step still removed every node whose source_file was in prune_sources, with no guard for re-extracted files — so a caller following the old edit-workflow (pass the changed file in BOTH new_chunks and prune_sources) had its freshly-built nodes deleted after the merge, silently losing a concept whose label survived the edit. Exclude new_sources (files present in new_chunks) from prune_set: a re-extracted file is being replaced, never deleted, so "replace" wins over a contradictory "delete" of the same source. Genuine deletions (in prune_sources but not new_chunks) still prune. Regression tests for both. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Switch every website URL from graphifylabs.ai to graphify.com — the hero logo, the Penpax section, and the waitlist link — across the main README and the translated READMEs. The contact email stays on graphifylabs.ai (mail is hosted there); no mailto links were changed. graphify.com is the official graphify product site; graphifylabs.ai remains the company site (org profile + email). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Graphify-Labs#1795) _reconcile_existing_graph treated "source identity absent from the collected corpus" as deletion and evicted its nodes/edges/hyperedges. But corpus absence is ambiguous: it's also what you see when a file still exists and merely stopped being collected (ignore rules or filters changed). Upgrading into the merged- .gitignore scan semantics (Graphify-Labs#1363) mass-evicted 655 nodes from a deliberately- built, .gitignore'd docs dir whose files were present the whole time — reported as a successful rebuild. Fail-closed: before evicting a corpus-absent identity, require Path(identity) .exists() is False (identity is an absolute path). Alive-but-excluded sources are preserved (nodes, edges, hyperedges) and a loud line reports how many were kept and why. True deletions and renames still evict (old path gone from disk); a full extract --force still purges deliberate exclusions via the AST ownership rule. Existence is memoized (one stat per file that left the corpus). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… twin (Graphify-Labs#1799) The semantic pass mints a document node <slug>_doc; the markdown quick-scan (extract_markdown) mints the bare <slug>. After a semantic build, a `graphify update` (AST path) re-runs the quick-scan and the graph ends up with BOTH — one document as two disconnected nodes, the file's edges split between them (semantic `references`/hyperedges on the _doc twin, quick-scan cross-links on the bare one). path/query traversals dead-end on the wrong twin; degree and communities split. build_from_json now reconciles the pair: when <slug> and <slug>_doc both exist with the same source_file and both are file_type=document, remap the bare node into the semantic _doc node (canonical, richer edges) and repoint its edges and hyperedges. Remap-induced self-loops are dropped; pre-existing ones are left alone. Gated to document twins for the same file, so a code symbol `foo` and an unrelated `foo_doc` never merge. Regression tests for both. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…bs#1797) querylog wrote every query/path/explain question + corpus path (and full responses under GRAPHIFY_QUERY_LOG_RESPONSES) to a default-on, unbounded, fail-silent plaintext file at ~/.cache/graphify-queries.log — outside any repo's .gitignore/retention, and undocumented. A default-on plaintext record of proprietary queries contradicts graphify's on-device / no-telemetry posture. Flip to opt-in: _log_path() returns None unless GRAPHIFY_QUERY_LOG_ENABLE=1 (default path) or GRAPHIFY_QUERY_LOG=<path> is set; GRAPHIFY_QUERY_LOG_DISABLE=1 still forces it off (back-compat, wins). Document all four env vars in the README (the old entries implied default-on). Regression tests cover default-off, both enable paths, disable-wins, and that log_query writes nothing without opt-in. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ify-Labs#1789) A solution folder is a virtual grouping, not a file: VS writes its name as both the display name and the "path" (name == path, no real file). extract_sln resolved it to an absolute filesystem path anyway and keyed the node id off that. The CLI id-relativization pass only remaps ids of real files in the scan set, so a virtual folder never matched and its absolute id (with the local username) survived into a committed graph.json. Detect solution folders (name == path) and key their id/source_file off the folder name only; real project files still resolve as before. Adds a regression test asserting the folder node id is relative. The earlier fix (0.9.13) covered .csproj/.sln file nodes but missed the virtual folders, so Graphify-Labs#1789 was closed prematurely; this completes it. Reported and diagnosed by @fremat79. Co-Authored-By: fremat79 <fremat79@users.noreply.github.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nvs (Graphify-Labs#1804) Graphify-Labs#1807 — piping graphify into a reader that stops early (head, Select-Object -First N, sed q) disconnected stdout mid-write, raising an unhandled BrokenPipeError (OSError(EINVAL) on Windows) and exiting 255, so CI wrappers and agent harnesses read a successful query as a failure. The console entry point now wraps the CLI body: a closed-pipe reader is treated as success — stdout is redirected to devnull so shutdown flush can't raise again, and the process exits 0. Adds a subprocess regression test. Graphify-Labs#1804 — .nox/ (nox virtualenvs, tox's successor, same .nox/ tree shape) was missing from _SKIP_DIRS while .tox was present, so nox site-packages got fully indexed (one repo came out 91% venv noise). Added next to .tox with a regression test. Reported by @varuntej07 (Graphify-Labs#1807) and @igorregoir-lgtm (Graphify-Labs#1804). Co-Authored-By: varuntej07 <varuntej07@users.noreply.github.com> Co-Authored-By: igorregoir-lgtm <igorregoir-lgtm@users.noreply.github.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-Labs#1810, Graphify-Labs#1809) Graphify-Labs#1810 — detection read only .gitignore/.graphifyignore, never .git/info/exclude, which is where git records local-only excludes and where `git worktree add` writes nested worktree paths. graphify walked into those worktree copies and the graph exploded (one 5-worktree repo: 9.4k nodes/10MB -> 210k nodes/311MB, ~77% duplicate). detect now loads info/exclude at lowest precedence (below every per-dir .gitignore, per git, so a nearer `!` still wins) and resolves the linked-worktree / submodule case where `.git` is a file to the shared common git dir. Graphify-Labs#1809 — two git-hook gaps: (a) post-checkout never honored GRAPHIFY_SKIP_HOOK, so the var stopped commit rebuilds but not branch-switch ones; now checked in both. (b) with core.hooksPath shared across worktrees, a commit in any linked worktree fired post-commit, which wrote a rogue delta-only graph.json into it and raced deploy/CI `git clean` against the detached rebuild. Both hooks now short-circuit in a linked worktree (git-dir != git-common-dir), comparing ABSOLUTE paths so the primary checkout (where --git-common-dir is the relative ".git") is never false-positived and skipped. Adds regression tests: info/exclude honored + negation precedence; both hooks honor the skip env and carry the worktree guard; and an end-to-end guard check against a real `git worktree`. Reported by @cdahl86-cyber (Graphify-Labs#1810, Graphify-Labs#1809); the worktree guard was co-developed with @Claude-Madera's PR Graphify-Labs#1806. Co-Authored-By: cdahl86-cyber <cdahl86-cyber@users.noreply.github.com> Co-Authored-By: Claude-Madera <Claude-Madera@users.noreply.github.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Graphify-Labs#1831 — `graphify export graphml` crashed on any dict/list-valued attribute (per-node metadata dict, graph-level hyperedges list) because nx.write_graphml only accepts scalars; a real ~2,300-node graph failed every export and left a 0-byte .graphml behind. to_graphml now coerces None->"" and JSON-serializes non-scalars across graph/node/edge scopes (int/float/bool/str pass through), and writes atomically via a temp file so a failed export can't leave a partial file. Closes Graphify-Labs#1830. Graphify-Labs#1807 followup — adopt @varuntej07's explicit in-guard sys.stdout.flush() from Graphify-Labs#1811: piped stdout is block-buffered, so a small fully-buffered output would only flush at interpreter shutdown (outside the guard), where a closed-pipe reader escapes as a noisy shutdown error and nonzero exit. Flushing inside the try closes that gap. Closes Graphify-Labs#1811. Reported by @hofmockel (Graphify-Labs#1831) and @varuntej07 (Graphify-Labs#1807/Graphify-Labs#1811). Co-Authored-By: hofmockel <hofmockel@users.noreply.github.com> Co-Authored-By: varuntej07 <varuntej07@users.noreply.github.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…#1757 followup) The Graphify-Labs#1835 fix scoped save_semantic_cache's final CLI write to an allowed_source_files allowlist, but the per-chunk incremental checkpoint in llm.py `_checkpoint_chunk` — the write that actually runs on every `graphify extract`/`update` via extract_corpus_parallel — still called save_semantic_cache with no allowlist. A chunk whose model result mis-attributes a node's source_file to another corpus file would merge that stray fragment into the victim's cache entry (merge_existing=True). Scope the checkpoint write to the chunk's own dispatched files (FileSlice -> .rel, bare Path -> the relative source_file). Also hoist the `import warnings` in cache.py to module level. Adds an extract_corpus_parallel integration test: a chunk dispatching only A.py that returns a node attributed to already-cached B.py must leave B.py's cache entry untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…phify-Labs#1766) Many nodes sharing one generic label (framework route handlers all labelled GET/POST, a repeated handler) consumed every BFS seed slot, so query traversal explored near-identical neighborhoods and buried the actual target. Seed selection now dedups by normalized label (GET/Get/get collapse together), keeping one representative per label, and the per-term guarantee loop honors the same cap so it can't reintroduce a dupe. Adopts @devcool20's seed-dedup from Graphify-Labs#1832 but drops that PR's second mechanism — a per-label multiplicity penalty applied inside the shared _score_nodes. That scorer also resolves shortest_path/explain endpoints, so dividing scores there silently reweighted path/explain (out of scope for Graphify-Labs#1766 and able to flip endpoint selection); the dedup alone bounds the flood. Also normalizes the dedup key (the PR keyed on the raw label). Adds the tests the PR was missing: dedup of homonymous labels, case/diacritic normalization, per-term-guarantee cap, and a guard that identical-label nodes still score equally in _score_nodes. Co-Authored-By: devcool20 <devcool20@users.noreply.github.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
function_types only recognised func/init/deinit/subscript, so computed properties (var body: some View { ... }) and willSet/didSet observers produced no node and their bodies were never walked — erasing the whole SwiftUI view layer. Emit a function-like member node for them and defer the body to the call-walk via function_bodies; stored properties are unchanged. Adds tests.
…ath forms macOS yields NFD paths from os.walk/getcwd while skill path literals are often NFC. Raw string compare treated every file as deleted+new and forced a full re-extract. Canonicalize at the manifest boundary (same idea as Graphify-Labs#1226). Fixes Graphify-Labs#2221
Stamped keys can already be NFC while Path(root).resolve() is NFD, so os.path.relpath treated in-root files as ../ and kept them absolute. Normalize both operands inside _to_relative_for_storage (and the join in _to_absolute_from_storage).
…bels Community labels are saved keyed by community id, but re-clustering reassigns those ids: after a rebuild that adds nodes, cid 30 can cover a completely different community and its saved name is then simply wrong. cluster-only already guards this — it validates each reused label against the `.graphify_labels.json.sig` membership fingerprints and re-hubs any community that changed (the case community_member_sigs() was written for). _rebuild_code() skipped that check entirely: it reused every label whose cid was present and hub-filled only the *missing* ones, so stale names survived — and then wrote them back to labels.json, laundering them as current. It also never refreshed the .sig sidecar, so the signatures kept describing an older clustering and drifted out of step with the labels they sit beside, leaving the cluster-only guard nothing accurate to check. Adding ~3.5k nodes to a real graph re-clustered 463 -> 515 communities and mislabeled 162 of them this way: a `domain.audit` namespace reading "ACH / bank payments", a `domain.auth` namespace reading "document-sensitivity.up.sql". Node and edge data stayed correct, so nothing failed loudly — only the names lied. Apply the same signature check in the incremental path, write the sidecar in step with the labels, and print the same "run `graphify label`" notice cluster-only emits so a drifted community set is visible rather than silent.
Crossing MAX_NODES_FOR_VIZ left the project with no graph.html at all: _rebuild_code unlinked the existing file and wrote nothing in its place. The delete on its own is defensible — a kept graph.html would describe an older, smaller graph — but the file is gone before the user reads the message, and the next incremental rebuild silently removes it again, so a repo that grows past the threshold just loses its visualization with no way to keep one. The export path already solved this (Graphify-Labs#1019): over the cap it re-renders the community-aggregation view rather than going without. Do the same here, so the artifact is current AND present instead of current OR present. GRAPHIFY_VIZ_NODE_LIMIT=0 still means "no HTML viz" (CI runners) rather than "aggregate", and if the aggregated render also fails the old skip-and-remove behaviour stands.
…o incremental targets canonicalize (Graphify-Labs#2211, Graphify-Labs#2213) The Graphify-Labs#2169 incremental canonicalization only rewrites edge targets that carry a target_file stamp. Python relative imports and markdown reference links emitted absolute-path-derived target ids without one, so on an incremental/subset extraction they dangled on an absolute id instead of resolving to the canonical root-relative node (dropping md->md references and leaving a dangling imports_from on --no-cluster). Both now stamp the resolved target (existence-gated); the stamp is popped before graph.json ships. Also register the unresolved target form in the remap loop so a symlinked root (macOS /tmp) can't cause an id-form mismatch. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Graphify-Labs#2212, Graphify-Labs#2210) Graphify-Labs#2212: benchmark, the merge-driver graph load, and callflow_html crashed or silently failed on a --no-cluster graph.json (edges stored under 'edges', not 'links'). A shared load_node_link_graph helper normalizes links/edges before node_link_graph and is used at all three sites. Graphify-Labs#2210: _stale_graph_sources compared graph source_file spellings to the scan with a raw string test (no NFC), and pruned any non-match with no liveness check, so alive files (macOS NFD paths, legacy basenames) were pruned as 'deleted'. It now compares NFC-on-both-sides and is fail-closed: a corpus-missing source whose file still exists is pruned only when the exclusion is provable, else kept with a warning. Prune message corrected to 'deleted or excluded'. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ng the global skill (Graphify-Labs#2215) claude_uninstall/gemini_uninstall/codebuddy_uninstall accepted project_dir but, with the default project=False, still deleted the user-global skill tree, so a library/test caller passing a project_dir nuked ~/.claude et al (the API trap behind Graphify-Labs#2168). They now take remove_user_skill and treat a passed project_dir as authoritative; uninstall_all opts in explicitly to preserve 'graphify uninstall' behavior. Also fixes a live CLI bug where 'uninstall --project' deleted the global codebuddy skill. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…phify-Labs#2221) Adds the regression test PR Graphify-Labs#2224 shipped without: an NFD-keyed manifest (portable and legacy-absolute) must match an NFC scan so --update is a no-op instead of re-extracting everything. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…#2213/Graphify-Labs#2210/Graphify-Labs#2212/Graphify-Labs#2215/Graphify-Labs#2221/Graphify-Labs#2181/Graphify-Labs#2218 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…Graphify-Labs#2206) _extract_python_rationale / _extract_js_rationale sliced the raw docstring/comment text to 80 characters before collapsing whitespace, so the cut could land mid-word, leave a run of literal spaces where a newline + indentation used to be, and, when the cut landed on a ".", produce an Obsidian export filename ending in "..md". Both _add_rationale sites now share _shorten_rationale_label, which normalizes whitespace first via textwrap.shorten (word-boundary safe, adds a placeholder only when it actually truncates) and falls back to a plain character truncation when shorten collapses to a bare placeholder -- which it does when the first word alone is already >= 80 chars (e.g. a comment opening with one long URL), a case that would otherwise regress to a content-free label.
…l args (Graphify-Labs#2241) walk_calls flattens an inline/untracked arrow or function-expression argument (one not separately tracked in function_bodies) onto the enclosing named function's caller_nid, so its calls resolve as if made directly by that function (Graphify-Labs#1630). But the closure's own parameters and locals were never folded into the shadow set used to guard argument-based indirect_call resolution, so a call argument inside the closure that happened to share a name with an unrelated callable elsewhere in the corpus produced a fabricated indirect_call edge, confidence 0.8 — even though the identifier was, in fact, a local binding one lexical scope down: rows.map((r) => c.get(r)) // `r` is the arrow's own param, not a // reference to some other same-named function Single-letter names make this common, since they collide with same-named symbols anywhere else in the repo (loop vars, test helpers). Fix: thread an extra_locals set through walk_calls's recursion. Entering an untracked closure folds that closure's own bindings (computed the same way as a tracked function's, via _js_local_bound_names) into extra_locals for its subtree only; deeper untracked closures compound the same way on their own recursion. All six call sites that build the caller's shadow set now union in extra_locals, so the fix applies uniformly to the argument, collection, and assignment/return capture paths already sharing that guard, not just the argument one that surfaced it. Tracked closures (const-assigned arrows, methods) are unaffected — they already get their own caller_nid and their own correctly-scoped shadow set. Scope: this fixes the shadow-set gap for closures. A `for (const x of xs)` loop variable not wrapped in a variable_declarator is a separate, pre-existing gap in the same shadow computation, already addressed by Graphify-Labs#1985 — not duplicated here.
…y-Labs#2052) self_type (`self: Logging with Database =>`, `this: T =>`) was never dispatched on anywhere in the Scala extractor, so a trait/class's structural precondition on its enclosing type produced zero edges, in any context. The type node sits at a fixed position among self_type's unnamed-field children (binder identifier first, type second when present), and _scala_collect_type_refs already handles every shape that position can take (type_identifier, compound_type for `with`, refinement bodies) -- reused unchanged, one new dispatch branch. Also add the new `requires` relation to DEFAULT_AFFECTED_RELATIONS, mirroring how `indirect_call` was wired into blast-radius traversal when it was introduced, so `graphify affected` follows it like the existing inherits/mixes_in/embeds structural relations. Covers: single type, `with`-compound, structural refinement (base type only, matching how refinement bodies are already unscanned elsewhere), the binder-only `self =>` shape (no requires edge), coexistence with an unrelated `extends`, and a plain class without a self-type (no spurious edge).
safe_name left stems like .env intact, so the vault wrote .env.md which Obsidian treats as a hidden file — invisible in the explorer and as unresolved wikilinks. Prefix with dot- (shared _obsidian_safe_stem for vault + canvas). True label stays in the note body. Fixes Graphify-Labs#2205
Assert .env / .gitignore become dot-env / dot-gitignore on disk and in the canvas file nodes, so Obsidian cannot hide them again (Graphify-Labs#2205).
…ge 2 Stage 2's .env regex treated .env.example / .sample / .template / .dist like live secret files and dropped them from the graph. Carve out those suffixes for .env / .envrc basenames only — real .env.local etc. stay blocked. Fixes Graphify-Labs#2184
…raphify-Labs#2243) Follow-up to Graphify-Labs#1899. That fix taught the relativization pass to catch a NODE whose id was minted from an absolute out-of-root path and give it a portable "ext_"-namespaced id, by matching the node's own id against _make_id(str(its source_file)). But several cross-file resolvers (Python relative imports, C/C++/ObjC quoted #include) only ever emit an EDGE for an import target, no node -- so when that target lives outside the scan root, the belt-and-braces pass has nothing to learn the old->new id from, and the edge keeps the raw _make_id(str(absolute_path)) slug forever. The scan path, including the OS username, ends up in links[].source/target, and differs between machines/checkouts even though the node id sets are identical. _import_c also never stamped the transient `target_file` hint (Graphify-Labs#1814/Graphify-Labs#2169) its Python/JS siblings already use for exactly this kind of cross-file target canonicalization, so it could not benefit from that machinery either. Fix, in the two places this root cause actually lives: - _import_c now stamps target_file on a resolved #include, mirroring _import_python/_import_js. - The id_remap pass that already walks target_file-stamped edges to canonicalize in-root-but-unscanned targets now also handles the out-of-root branch it previously skipped ("leave its ids alone"): an existing out-of-root target gets the same portable ext_-namespaced id an out-of-root NODE already gets, so an edge with no node of its own is covered too. A target that does not exist on disk still stays dangling, unchanged from before. _portable_out_of_root_sf moved next to id_remap so both the new edge-target branch and the existing node-level pass share one implementation. Four tests in tests/test_extract.py: the out-of-root #include gets a portable id instead of the raw slug, and its transient target_file hint never leaks into the returned edge; the same corpus built from two differently-nested checkout paths produces a byte-identical target (the reported non-determinism, made explicit); an in-root, same-batch include still resolves to the real node's id (negative/regression guard); and the equivalent out-of-root Python relative import is fixed too, since the gap was in the shared remap path, not language-specific. Known limitation: this covers every current target_file-stamping resolver (Python relative imports, C/C++/ObjC #include, JS/TS/Svelte/Astro/Vue rescued imports). A resolver that mints a path-derived edge target WITHOUT stamping target_file at all -- none do today -- would still leak; the fix closes the gap in the shared mechanism, not a per-language allowlist.
…bs#2231, Graphify-Labs#2243) Absolute/machine-slug ids still leaked into edge endpoints from producers the target_file-stamp loop didn't reach. Three fixes: apply id_remap to raw_calls caller_nid so module-top-level indirect_call sources canonicalize (Graphify-Labs#2231); a general backstop in the final relativization pass that learns _make_id(abs source_file) -> canonical id for every node and rewrites all node ids and edge endpoints (in-root -> _file_node_id, out-of-root -> ext_), suffix-aware for __entry; and target_file stamps on bash source/entry edges so they ride the same canonicalization. No node id or edge endpoint now carries the scan-root slug for any file in the batch. Builds on Graphify-Labs#2250. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ook rebuild (Graphify-Labs#2251) _reconcile_existing_graph loaded graph.json inside a swallowing try, so a graph that was merely unreadable (over the size cap or unparseable) was silently replaced by the code-only extraction, in both the clustered and --no-cluster hook paths (force made it worse). It now loads through the fail-closed build._load_existing_graph and _rebuild_code refuses the write (prints and returns False) on a load failure, matching the CLI path; the --no-cluster write is now atomic with a protected-graph backup. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
….example regression test (Graphify-Labs#2205, Graphify-Labs#2184) Follow-ups on the cherry-picked Graphify-Labs#2242/Graphify-Labs#2232: an all-dots label ('...') no longer produces an empty 'dot-' Obsidian stem (falls back to 'unnamed'), and the .env.example carve-out gets the regression test it shipped without (templates graphable, real .env still sensitive, secrets/.env.example still dropped). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…#2243/Graphify-Labs#2251/Graphify-Labs#2241/Graphify-Labs#2253/Graphify-Labs#2206/Graphify-Labs#2184/Graphify-Labs#2205/Graphify-Labs#2052 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`_score_query` matches a question against `norm_label`, `label_tokens`, `source_file` and the node id — never against `rationale`. But the extraction spec deliberately stores WHY a decision was made as a `rationale` *attribute* on the concept node rather than as a node of its own, so for a "why does X ..." question that text is frequently the only place the question's wording occurs. The node that holds the answer is therefore reachable only by someone who already knows the label it hangs on — which is what the asker does not know. Adds a rationale tier at 0.75: below the label substring tier (1.0), since the node's own name remains the stronger signal, and above the source-path tier (0.5), since rationale is authored prose about the node while a path fragment is incidental. Like the source tier it does NOT count toward term coverage, so it adds recall without ever letting a long rationale win back an exact-label tier it did not earn. `_node_search_text` gains the same text so the trigram prefilter stays a complete candidate generator — without that, a node the scorer would match could never become a candidate. `_find_node` is unaffected by design: finding a node *named* X must not surface nodes merely discussing X, and the widened candidate superset is re-filtered by its own predicates. Measured on a 5,524-node corpus, five natural-language "why" questions whose answer node carries a `rationale`: answer's rank in the ranking 126->2, 278->14, 37->3, 5->4, 4->4 answer reached by the query 0/5 -> 2/5 The remaining gate is seed selection, not scoring: the answer now ranks 2nd-4th for four of the five questions but `_pick_seeds`' gap cutoff still drops it in three. That is left alone here as a separate, more opinionated change. Six tests added; three of them fail on the unpatched scorer. Full suite shows no new failures (the 44 pre-existing Windows/optional-dep failures are identical before and after).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2293.
_score_querynever readsrationale, but the extraction spec stores the WHY as arationaleattribute on the concept node rather than as a node of its own. For a "why does X …" question that prose is frequently the only place the question's wording occurs, so the node holding the answer is reachable only by someone who already knows the label it hangs on.What changed
_node_rationale_text(data)— one helper, normalized like a label, tolerant of a list-valued attribute. Single source of truth so the scorer and the index cannot drift._score_query— a rationale tier at0.75: below_SUBSTRING_MATCH_BONUS(1.0), because the node's own name stays the stronger signal, and above_SOURCE_MATCH_BONUS(0.5), because rationale is authored prose about the node while a path fragment is incidental. It is folded into the singleton score too, so per-term seeding sees it._node_search_text— same text added, so the trigram prefilter stays a complete candidate generator. Without this a node the scorer would match could never become a candidate.Deliberately not counted toward
matched(term coverage), mirroring the source tier: coverage scales the exact/prefix tiers, and a long rationale mentioning several query words must not win back an exact-label tier it did not earn. The change adds recall, never precedence._find_nodeis unaffected by design. Finding a node named X must not surface nodes merely discussing X. The widened candidate set is a superset that_find_nodere-filters with its own predicates, so its results — and their order — are unchanged.test_find_node_ignores_rationalepins this.Tests
Six added. Three fail on the unpatched scorer (verified by reverting
serve.pyand re-running):test_score_nodes_matches_rationale_attribute— the behaviour itself. The fixture is built so no content word of the question appears in the label, id or path, and the test asserts that isolation inline; an earlier draft passed without the patch via an incidental label hit and proved nothing.test_score_nodes_rationale_does_not_count_toward_term_coveragetest_node_search_text_includes_rationaleThree more are regression guards that pass either way: rationale never outranking a label match, the prefilter/full-scan equivalence extended to rationale text, and
_find_nodeignoring it.Suite
tests/test_serve.py+tests/test_query_cli.py: 137 passed.Full suite: 3597 passed, 44 failed — the same 44 as on an unmodified checkout (
diffof the twoFAILEDlists is empty). They are Windows/optional-dependency failures intest_skillgen,test_terraform,test_watch, etc., unrelated to this change.Effect, measured
5,524-node corpus, five natural-language "why" questions whose answer node carries a
rationale:Honest about the ceiling: scoring was one of two gates. The answer now ranks 2nd–4th for four of five questions, but
_pick_seeds' gap cutoff still drops it in three. Seed selection is a more opinionated change and is left out of this PR — noted at the end of #2293 so the partial result is not mistaken for the whole fix.