Repository navigation
feat: port the colby v1.6.2 extraction, resolution and sync semantics - #322
Merged
Merged
Conversation
equivalence.rs only proved that each committed colby.db agrees with its own JSON; nothing in the suite ran the extractor, so a change that moved extraction output passed CI until someone ran the out-of-repo regeneration helper by hand. - tests/golden_reextract.rs indexes every re-indexable corpus from crates/codegraph-bench/fixtures/ with the binary under test, dumps it through the same canonicalizer as `bench --gen-golden`, and compares the five artifacts byte for byte with reference/golden/. A failure names the changed artifacts, the canonical row diff and the regeneration command. A guard test keeps the corpus list in step with reference/golden/. - scripts/regen-goldens.sh runs the shared recipe: --check (writes nothing), --write <corpus...> (only the named corpora) and --mini-transplant (mini's schema rebuild). Scratch directories come from mktemp and are removed on exit. - equivalence.rs generates each corpus's two byte-drift tests from one macro, keeping the existing test names, and fails when a committed corpus has no test pair. - docs/equivalence.md describes the recipe once, adds the regeneration log the v0.55.0 program records changes in, and each corpus section now says what it guards plus the one regeneration command. Every corpus re-extracts byte-identically on this tree; a tampered go/nodes.json makes the gate fail with a Tier-1 node diff.
…s and CommonJS corpora The goldens had no corpus for C#, Java, Swift, Objective-C, PHP, Vue or Svelte, and the Python corpora had no class with bases, so whole families of extraction never reached the equivalence oracle. Each new corpus is a small source tree under crates/codegraph-bench/fixtures/ with its golden under reference/golden/, generated by `scripts/regen-goldens.sh --write`. They capture today's output on purpose, misses included, so the corrections that follow show up as reviewable diffs attributed to one change each. Both golden suites now cover them, and docs/equivalence.md says what each one exercises and logs the addition.
Four multi-project corpora give the framework resolvers and synthesis passes a baseline before they land: routers (React Router, Next.js, Expo Router, Vue Router, Nuxt, SvelteKit, Angular, TanStack), servers (Express, NestJS, Flask, Django, FastAPI, Spring, Laravel, Rails, ASP.NET, Vapor, axum, actix-web, Gin, Play, Drupal), mobile (React Native and Expo native modules, a Swift/Objective-C bridge) and synthesis (emitters, handler tables, re-renders, stores, Go embedding, C function pointers, Kotlin expect/actual, MyBatis, Celery, Spring events). Each project carries the manifest detection reads, and projects share symbol names on purpose, so the cross-project misses in today's output (an Angular @component bound to a React file's Component) are in the golden and their later removal shows up as a diff.
…n version to 22 Schema 9 adds the `synthesis_inputs` table and the partial `idx_edges_synthesis_site` index the synthesis stage uses to decide when a sync must refresh synthesized edges. The extraction version moves once, from 21 to 22, ahead of every graph-semantic change in this release, so an index written by v0.54.0 reads as outdated and is rebuilt exactly once. Every golden corpus regenerates only `schema.sql` and `colby.db`; the canonical JSON is unchanged (golden_reextract proves it), and mini is transplanted with `scripts/regen-goldens.sh --mini-transplant`. `upgrade_from_v054` plants a real v0.54.0 index (fixture built by the released binary, provenance in its README) and upgrades it through both `codegraph sync` and the daemon catch-up sync, asserting schema 9, the extraction stamp 22 and a canonical graph equal to a fresh init.
Add the `probes` test binary (crates/codegraph-cli/tests/probes). A probe writes a small tree into a temporary directory, runs the built `codegraph init` with CODEGRAPH_NO_DAEMON=1 and CODEGRAPH_NO_WATCH=1 so scanning, extraction, framework detection and resolution run as they do for a user, and reads the published index back through an immutable=1 SQLite connection, which leaves no sidecar for a later sync to refuse. `support.rs` provides the Project builder, edit/sync helpers that stamp explicit mtimes instead of sleeping, row queries over nodes, edges, unresolved references and files, a selector notation shared with the failure output, and the assert_edge!, assert_no_edge! and assert_unresolved! macros, whose failures list the project's whole non-contains edge list. Smoke probes pin a TypeScript import call, an unresolved external import, a Python method call and a sync re-read.
Add the `sync_convergence` test binary. Each case indexes a `before` tree with the built `codegraph init`, applies one edit (add, modify, delete or rename files), and syncs it through two drivers: the CLI `codegraph sync`, and the watcher path, `sync_changed_paths` with exactly the changed paths (or the full reconcile the real watcher runs for a control file). A fresh `init` of the final tree in a separate directory is the oracle; both drivers must match it under `canonicalize_db` + `diff_canonical`, by full natural keys. Cases name the edges their before and fresh indexes must hold, the on-disk trees must equal the oracle's, and failures print a readable row diff. The cases cover every language with an extractor and every change kind: an added import target, a body-only edit (#2288 same-named methods and Java overloads), a rename, a deleted module, a heritage edit, a same-named competitor, control files (codegraph.json, config.toml, .gitignore, tsconfig.json, package.json) and a C header that turns into C++. Gates fail when a language with an extractor or a change kind has no running case. Known gaps are ignored with their reason and fail under --ignored: W5-01 #2392, W5-02 #2403 and W5-03 #2450, plus four unscheduled gaps this matrix found: an Erlang remote call and a MyBatis cross-mapper include are not retried when their target appears, and a tsconfig.json `paths` or package.json framework change does not re-resolve unchanged files. The probe `file:` selector now matches a file node by path, since the embedded extractors key theirs by a hash.
The live-events test once held a Windows CI job for 40 minutes: the sync by another writer and the final search had no time bound, and the #[tokio::test] runtime's drop waits forever for a stuck spawn_blocking task, which the viewer uses for every request. Each test now runs on its own runtime through `run`, which bounds the test body, propagates a panic from it, and shuts the runtime down on a helper thread with a limit of its own. Every step that can block has a named limit: indexing the fixture, starting the viewer, connecting to /api/events, each awaited event, each request, the sync by another writer, closing the hub, dropping the last client and the teardown. Blocking steps run on the blocking pool so a stuck one never stalls the runtime. A hang fails with the step's name (or, for the backstop, the last step started). No assertion changed.
`bench --ab` times a cold `init` of a baseline and a candidate codegraph binary on pinned corpora. Every run copies the corpus into a fresh project without `.git` or an index directory, sets RAYON_NUM_THREADS (the CLI has no thread flag), CODEGRAPH_NO_DAEMON and CODEGRAPH_NO_WATCH, removes CODEGRAPH_DIR, and records the wall time, the wait4 peak RSS and the canonical fingerprint of the database. Runs alternate between the binaries and the first run of each cell is a discarded warm-up. The harness stops without a report when a run fails, is killed or times out, leaves no database or one that fails PRAGMA quick_check or canonicalization, or when one binary produces two different graphs at any thread count. The JSON report records the CPU, kernel, rustc, both binaries' SHA-256, every run, median and MAD, candidate/baseline ratios and the regression-budget verdict; --report-md renders it as a table. `bench --graph-diff` compares two canonical graphs, each a database or a golden directory, and prints the removed and added rows grouped by surface, language, kind and resolvedBy. It exits 0 when the graphs are identical, 1 when they differ and 2 when an input cannot be loaded. A database is read from a private copy, so the input never gains SQLite sidecars. The corpus registry gains express-medium and the release-pinned leveldb, gson, Newtonsoft.Json, Alamofire, redis and kong corpora, a tag field, and --corpora-root for --list-corpora and --fetch-corpora. sha2, already in the lock graph through codegraph-core, hashes the binaries and graphs. docs/benchmark.md documents both commands, the report fields, the regression budget and every pinned corpus.
V1_6_2_AUDIT_2026-10-10.md records the 268 upstream commits from colby v1.6.1 to main of 2026-10-07 row by row: disposition, Rust target, the v0.55.0 program commit that carries each row and its status, together with the Rust-only defects and pre-existing extraction gaps the audit found and the owner decisions of 2026-10-10. COBOL is OWNER-EXCLUDED, and refusing a newer schema is recorded as PLANNED for its own follow-up pull request. The ledger's discovery boundary moves to b635dd46, and docs-check follows it; the tracked release stays v1.6.1 until the ported rows ship.
…egistry `index --force` lists every registered stdio MCP server on stderr before it rebuilds. The pre-heal ordering test drains the child's pipes only after both lease barriers, so on a machine with many registered servers, and pipes capped at two pages once the user's pipe budget is spent, the list filled stderr and parked the command before its second lease. The test now gives the command an empty registry of its own.
The single extraction-version move to 22 must be exactly one step past the version the preceding release wrote, which is 21 for v0.54.0.
Codecov Report❌ Patch coverage is ❌ Your patch check has failed because the patch coverage (94.15%) is below the target coverage (95.00%). You can increase the patch coverage or adjust the target coverage. @@ Coverage Diff @@
## main #322 +/- ##
==========================================
+ Coverage 95.06% 95.60% +0.54%
==========================================
Files 198 280 +82
Lines 109293 166327 +57034
==========================================
+ Hits 103895 159012 +55117
- Misses 5398 7315 +1917
... and 89 files with indirect coverage changes 🚀 New features to boost your workflow:
|
The upgrade test built its immutable URI from the raw path, which on Windows keeps the \\?\ verbatim prefix and backslashes that SQLite cannot open. It now uses the same file: URI form as the probe harness.
dump_golden wrote its log beside the output directory, so --write left reference/golden/<corpus>.gen.log next to the named corpus and removed that path on exit. Both logs now live in scratch directories of their own.
…s W0-06 requires The corpora lacked three shapes the plan asks them to pin at v0.54.0 semantics: a PHP `static::` call, a CommonJS `require().member`, and a Flutter app whose `pubspec.yaml` triggers detection. Each lives in a new file, so the regenerated goldens only add rows: `bench --graph-diff` against the previous goldens reports 0 removed rows for all three.
The UI-family, extraction-gap, carry-forward, planning-finding and Rust-only-defect tables now carry the disposition and Rust target columns the ledger format requires, and the not-needed section is a row ledger.
--mini-transplant still logged to a path beside its scratch directory and removed it on exit; the log now lives in a scratch directory of its own, like the other two.
…ture The script-tests loop ignored a failing script, so ci-gate.test.sh had silently stopped at its stale needs anchor since the UI job joined CI Success (#298): mutations C to K never ran. The anchor now carries the ui job and the loop stops on the first failing script; all five harnesses pass.
…tarts A cold `serve --mcp` answers its handshake direct and spawns the shared daemon fire-and-forget, but never connected to it. With no client the daemon idle-exited after CODEGRAPH_DAEMON_IDLE_TIMEOUT_MS (300 s by default) and the session's index silently stopped syncing. The session now keeps a passive lease on that daemon from a background thread, the same client connection an explicit projectPath service holds, so the handshake still never waits on the daemon. When the lease stops being live (the daemon died, was replaced, or the index was re-created) it starts or re-attaches a daemon with backoff: CODEGRAPH_DAEMON_RETRY_MS (default 5000, 0 disables) doubling up to CODEGRAPH_DAEMON_RETRY_MAX_MS (default 300000), as upstream #2293 does for its degraded proxy. It never starts a daemon for a project whose index is gone.
A settled burst used to sync exactly the paths its events named, however many, and the notify callback forwarded only the event kind and paths, so a backend's sign of dropped events was lost and the index could stay stale after a mass rename or a branch switch (#269). The watcher now runs one full project reconcile instead when the burst named more than 500 paths (upstream #1397's SCOPED_SYNC_MAX_PENDING), when an event carries notify's Rescan flag (an inotify queue overflow, an FSEvents drop) or names no path (upstream #2025's filename == null), and on Windows, whose backend can lose events silently, when an idle drift sentinel (CODEGRAPH_WATCH_SENTINEL_MS, default 60000, 0 disables) finds changes no event reported; it then also registers the root watch again. Until a lost-event reconcile commits, the watcher reports RECOVERING with the cause, and the RECOVERING banner now names its reason instead of assuming lock contention. As promised on #269, the MCP instructions, the codegraph_status description and the skill now say what happens after a large burst and that renames never need a rebuild; the initialize golden follows the instructions.
A daemon keeps running the code it started with. After an upgrade the project's daemon could be an older build than the session that found it: the proxy saw a version mismatch, the session fell back to serving reads, and the old daemon kept the writer slot for good (upstream #2343, #2346). A session that meets a daemon of another version now reads its version from the hello and replaces it only when it is an older plain X.Y.Z release: it asks it to drain over the project-bound control channel, so nothing is signalled by pid, then starts a daemon from its own install and proxies to it, or attaches through the cold-start lease keeper if the new daemon is slower than the usual poll. A newer release, a prerelease, an unknown version, or a daemon that does not acknowledge the drain is never stopped; the session then serves reads and says so on stderr. Every daemon also records its executable at start and drains and exits once the file is replaced or removed (CODEGRAPH_DAEMON_INSTALL_CHECK_MS, default 30000, 0 disables), and a cold-start session whose own install changed stops starting daemons. A test-hooks feature lets test builds of the daemon claim another version and leave control frames unanswered.
A content-hash comparison calls a file up to date even when its stored row holds none of its symbols, so `status` said "Index is up to date" while a file contributed nothing to the graph (upstream #2336, the status half of #2346). `status --json` adds `index.filesNeedingReindex` (rows stored with no nodes and no recorded reason, which a hash-based sync never revisits) and `index.filesWithParseErrors` (a recorded parse failure and no symbols from it); files over the size limit and file-level-only languages are empty on purpose and count in neither. Human output names up to three files of each group and no longer calls such an index up to date. `files --json` carries each file's recorded errors. Every existing field is unchanged.
`serve --mcp` launched in an unindexed directory with `.git` searches downward for indexed sub-projects. The walk had a depth and candidate cap but no limit on how much it read: launched in /tmp (~40K entries) it never answered `initialize` within 30 s. The scan now stops after 10,000 directory entries (counted, so where it stops is deterministic) or 500 ms, whichever comes first, and a scan that stopped early adopts nothing, since it cannot know the child it found is the only one; the tool guidance and the stderr notice say it stopped. The MCP servers no longer run the scan in their constructors, before `initialize` is answered: their first `tools/list` or tool call runs it, and reports the candidates it finds once.
The daemon runs detached, with no console of its own, so on Windows every console program it starts without CREATE_NO_WINDOW opens a visible console that flashes once per call. The git pending-status path already set the flag; `git_command` (repository and hook discovery) and the viewer's `git config user.name` did not (upstream #2280). `codegraph_watch::hide_console_window` now sets the flag in one place, and every git call, and the viewer's browser opener, goes through it. Because std cannot report a command's creation flags back, a static source guard (like upstream's) checks every `Command::new(` in the crates the daemon runs: each must hide its console or say in a `console-window-guard:` comment why it needs not.
KNOWN_DIFFS listed C++ receiver-type inference, Java/Kotlin field-receiver inference, the batched and conformance passes and the per-language import refinements as deferred, though all of them are ported. equivalence.md said `ns::compute()` resolves through the qualified-name matcher, while the cpp golden records it as an exact-name match, and called the missing Rust struct-literal instantiation upstream's own shape, which it is not. A dated UPSTREAM.md entry records the corrections, the KEEP-RUST status of #226's stricter receiver-word gate, and the #1728 disposition; the entries it corrects stay as written.
- Read <root>/.git/info/exclude ahead of the root .gitignore in RootGitignore, so the scan and the watch policy prune what git's exclude-standard prunes (upstream #1728). A .gitignore rule wins a conflict, as in git. Only a real .git directory at the project root is read, through the bounded stat-first reader; a .git file, a link, an ancestor repository and core.excludesFile are never followed. - The watcher treats the exclude file as a control file: per-directory backends watch .git/info, and an edit reloads the scope and runs one full reconcile. - The pending-status git record's scope fingerprint includes the exclude file, so a record made before an edit is not trusted after it. - Sync matrix: a .git/info/exclude control-file case. The root .gitignore case now hides legacy/ rather than vendor/, which the default ignores already drop, so it moves an edge instead of passing vacuously.
- An import node is a statement, not a definition (upstream #965, 0a91d0f5). Exact name matching no longer offers one, the file's own or another file's, and one named like a definition no longer makes that definition look ambiguous. The fuzzy fallback already considers callables only. The qualified-name matcher is unchanged here: its imports-to-own-statement edges belong to the #2392 target gate (W3-01). - Goldens: csharp, erlang, java, mobile, php, python, python_bases, routers, ruby, rust, servers, swift, synthesis, typescript, vue lose every imports edge exact matching bound to an import node, and the refs stay unresolved. PHP `use App\Field;` now binds namespace App\Field; Ruby `require_relative "logger"` falls through to the fuzzy match class Logger. Logged in docs/equivalence.md. - Probe pr0965_vue_import_never_binds_a_svelte_import_node, unit tests for the exact and fuzzy paths, and a PHP AddCompetitor sync-matrix case. - tier2_false_edges: the phantom node:path import now binds nothing and stays parked as unresolved (renamed to phantom_import_binds_nothing_and_stays_unresolved); the Rust supertype fixture keeps only its in-repo `use` edge, its `crate`/`std` self-edges gone.
- The out-of-repo import gate (imports and supertypes) no longer counts a JS specifier as local because it starts with a path alias's prefix: a catch-all "*" alias has an empty prefix that every package starts with (upstream #2437, 2c807f5c). An alias now makes the import local only when resolve_import_path maps it to a file the index holds, so an alias landing in node_modules or on a directory does not either. Resolving an import still honors the prefix, so "*": ["src/*"] aliases resolve. - ResolutionContext gains is_indexed_file, answered from the store or the snapshot's known files, never from the filesystem. - Probes pr2437_catch_all_alias_keeps_package_imports_external and pr2437_alias_into_node_modules_keeps_the_import_external, unit tests, and a Tsx AddCompetitor sync-matrix case; the matrix gains no_edge_after for an edge a fresh index must not hold. - No golden changes (scripts/regen-goldens.sh --check: 32 identical).
- detect_language_for_source (and detect_language_of for a bounded read) give a file's language as extraction parses it: a .h header holding C++ or Objective-C is that language. index, sync, migration and framework extraction record and use it, so a header's file row agrees with its nodes (G15). Framework extraction reads a C header whenever a resolver applies to C, C++ or Objective-C. - Goldens: cpp plain_derived.h and ue_actor.h become cpp, objc Sources/Base.h becomes objc; files.json only. Logged in docs/equivalence.md. - c_family_correctness pins the sniffed language for headers and the agreement of every node; the c_header_flips_to_cpp sync-matrix case now requires the fresh index to record shape.h as cpp, via the new file_language_after expectation.
Claude Code hands a subagent's report back to the parent session as a UserPromptSubmit prompt wrapped in <agent-message from="...">...</agent-message> (upstream #2184, #2281). The hook took it for a user question and ran a full explore over the whole report. A prompt that is entirely that envelope, its opening tag bare or followed by whitespace and attributes, now exits before any index work, by the same anchored rule as the task-notification envelope; a question that mentions the tag, or follows the envelope, still runs. The CLI and MCP docs now describe both envelopes as the whole prompt.
On Windows a port another program holds exclusively (SO_EXCLUSIVEADDRUSE), or one inside a reserved range, fails the bind with a permission error, not as in use, so `codegraph ui` stopped at the first such port and blamed the privileged-port rule for ports like 4747 (upstream #2299, #2353). The fallback walk now moves past a port in use on every platform and past a refused one on Windows at any port and elsewhere from 1024 up; below 1024 off Windows a refusal stays the privileged-port rule and the walk stops with that message. The messages say what is true where they are read: Windows names the exclusive hold or reserved range and the netsh command that lists the ranges, POSIX says the system refused permission above 1024, and a walk that ran out says "in use or reserved" or "in use or not allowed" when a port was refused. Unit tests drive the walk through a fake binder for both platforms.
…d routes A directed Flow's from and to went through the free-text token filter, which drops a two-letter name, a Ruby predicate or bang method and an Objective-C selector, so the other end was then reported as naming nothing (upstream #2262). Directed mode now takes the two names as written. The named walk kept whichever route reached a node first, so of two routes of one length it followed the callee the index listed first even when the query named the other (upstream #2453, the half the viewer's Flow uses; explore's own Flow is W7-06). A node reached again from the same depth now takes the route that ends on fewer unnamed hops, then the one through more named symbols; of two equally deep named ends a seed keeps the one whose route passes more named symbols. The expansion order and the visit cap are unchanged. Upstream met it through an interface method's implementations, whose dispatch edges come with the synthesis stage (W7), so the tests use plain call routes, each asked naming one route and then the other.
Port upstream #2102 and #2270. A folder with exactly one subfolder and no file of its own, read from every indexed file with tests included, no longer counts as a level of the Map: a Maven or Gradle project's src/main/java/com/acme/app/ chain is one level, so the map opens on the packages instead of one box. The module's wire label writes such a chain of three or more folders first/…/last; the canvas box shows and is sized for the label, while the id, tooltip and side panel keep the whole path. A renamed lone (root files) bucket keeps its folder's elided label. The default root now counts files loose in the repository root in the whole, and when that root draws one box at every depth while the whole repository draws more, the Map opens on the whole repository. The SVG export writes the label too. Upstream's export still writes the id cut to the narrower box; docs/ui.md records the difference, and the non-production folder rule this port has not taken from upstream.
A Map box is border-box, so its 1px border (1.5px when selected) sits inside the width, but the width allowed only the 44px and 18px insets. A box whose label decides its width clipped the last character to an ellipsis: `src/main/…/clinic/(root files)` needed 216px and had 214.7px. Both lines now get the insets and a selected border.
Port the non-Steps half of upstream #2264. The viewer keys its lists by id, and a repeated id stops the page drawing. The routing manifest is a row per (route, edge), so a route bound to two symbols, or an inline handler whose every call reads as a handler, came back more than once. The routes list, and the routes among the entry points, now fold the rows to one per route: the row naming the route's root, else an `inline handler` row standing for a handler written at the registration, which opens on the route itself and offers no Flow. The root is upstream's route-roots order of evidence (a named handler, a rendered component, the route's own calls), with ties broken by line and id so the answer does not depend on edge order. Rows are over-fetched four to a route so folding still fills the page, and each row carries `inline`. A symbol's members list each member once and never the symbol itself or a file, the guard upstream added for components that contained their own script symbols twice. This index records neither repeat today; the resolvers here bind each route to one handler and mint no inline ones.
Port upstream #2405. The Dead code list showed two kinds of method a framework calls. A method whose decorator comes from outside the index (`@HostListener`, `@Cron`, `@OnEvent`) has no `decorates` edge and no decorators on the node, only the resolver's unresolved `decorates` row. That row now counts as a decorator, the same evidence the ancestor rule reads for an unresolved `extends` or `implements` row. A lifecycle hook on a class that skips the `implements` clause (`ngOnInit` on a `@Component` without `implements OnInit`) had no evidence at all. A new `hooks` rule drops a TypeScript method when its class carries a decorator and an interface outside the index carries the name, going by the classes that write the interface out: every one has the method, declared or inherited, at least two and more than half declare it, and more than half of them carry a decorator. TypeScript only, because the clause is erased there and the framework can only find the hook by name. The ancestor walk moves out of override_candidates into ancestors_of, shared by both rules, and the store gains the two unresolved-reference lookups the rules read.
Upstream d6efd43 (#1306) made one switch honor NO_COLOR and --no-color and go plain when stdout is piped. This port prints no color of its own and draws progress only on a terminal stderr, so its piped output is already plain whatever the environment says. Pin that, case for case with upstream's test: list commands under no signal, NO_COLOR with FORCE_COLOR, FORCE_COLOR, CI and CLICOLOR_FORCE; piped index and sync with no carriage-return rewrites; and NO_COLOR winning over CLICOLOR_FORCE for clap's help and errors, the one colored surface. docs/cli.md says so, and that upstream's --color and --no-color are not flags here.
…erver does
Claude Code 2.1.294 and later speak MCP 2026-07-28, which has no
initialize: a session opens with server/discover and names its protocol
version in every request's _meta, and a 2026-07-28 result carries
resultType "complete". The daemon proxy answered every tools/list itself
as {"tools": ...}, without resultType, and the client refused the tools.
Cold sessions serve direct, so only the second and later sessions of a
project, the ones a running daemon serves, broke.
The proxy now answers tools/list locally only in the initialize
lifecycle: after an initialize, before any server/discover, when the
request names no protocol version in _meta. Every other tools/list is
forwarded and owed in the reply ledger, and the daemon's rmcp handler
answers it as a direct server would.
The local initialize answer also pinned 2024-11-05, where the direct
server echoes a requested version that has an initialize handshake
(2024-11-05 through 2025-11-25) and falls back to 2024-11-05 for
2026-07-28 or an unknown version. It now negotiates by that rule, read
from rmcp's KNOWN_VERSIONS and pinned to rmcp's own negotiation for the
handler by a test.
A CLI test drives a proxied session (a warm serve --mcp while the
project's daemon runs) and a direct one through server/discover and
tools/list, and through initialize at 2025-06-18, 2024-11-05, 2026-07-28
and an unknown version, and compares every reply but its id; it fails on
the previous proxy.
Port the classifier half of upstream 950686de. A folder named `integration`, `sample(s)`, `example(s)`, `fixture(s)`, `benchmark(s)` or `demo(s)` anywhere in a path made a file non-production, so spring-petclinic's `src/main/java/org/springframework/samples/petclinic/` ranked as test code and the Map drew every one of its packages as a test. Only the project layout above the first `/src/` counts now; the package path below it never does. The narrow test predicate is now the one the resolver's test-suite rule reads, `codegraph_core::file_class::is_test_path`, instead of an older copy of it. That copy lacked two things upstream's `isTestPath` has: test-support folders (`*-test` modules, `testdata`, `testutil(s)`, `fakes`, `mocks`, `__mocks__`, `stubs`, from 950686de) and an `e2e/` tree (from b1f40c57). The classification is read at query time only, by search ranking, dead code, named-symbol flow and the viewer's test flags; the graph and the extraction version do not change. The scoring tests now carry upstream's `is-test-file.test.ts` case for case, and the Map and dead-code fixtures go back to upstream's petclinic and `demo` package paths.
`serve --mcp` at `$HOME` or the filesystem root runs no daemon, watcher or catch-up, and serves through the hand-rolled loop that can hand the session to a daemon once the client's roots name an indexed project. That loop knew only the `initialize` lifecycle. A 2026-07-28 client's `server/discover` got "Method not found", its `tools/list` came back without `resultType: "complete"`, and an `initialize` was answered with `2024-11-05` whatever version it asked for. A session that opens with the inline lifecycle, `server/discover` or a request naming its protocol version in `_meta` before any `initialize`, now ends the loop unanswered. The rest of it is served by the rmcp handler the direct server uses, built from the state the loop already resolved: default project, cwd, root discovery and the explicit-`projectPath` service broker. It still starts nothing for the broad root. The lifecycle has no roots, so such a session never adopts the client's workspace; a call that passes `projectPath` gets that project's daemon and watcher as before. Each `initialize` reply branch, plain, with a `roots/list` request, or adopting the client's root, now negotiates the version the rmcp handler would. A CLI test drives both lifecycles through a session at a temporary `$HOME` and a direct one and compares every reply.
A sweep of the pages against what v0.55.0 changed. The environment variable table lists CODEGRAPH_MCP_TOOL_TIMEOUT_SECS, the per-tool-call limit an MCP client waits on (60 s, 0 for none), which no page named; the daemon retry, install check and Windows sentinel variables were already there. The skill quotes the stale-index banner as the server prints it, a heading line and one indented path per file, not the single-line form it had. README and its Chinese mirror agree in headings, commands and links.
Every PLANNED row of the v1.6.2 audit now records what happened to it. A row the v0.55.0 pull request carries is LANDED with the commits that carry it; a gate fix belongs to the row it fixed, and a row naming upstream PRs cites the commits that port them. A row under W7, W8 or W9 is PLANNED (follow-up PR) since the owner split the program on 2026-10-16 (OD-18), and a row that lands only partly names both halves. Where the Corrections table moved a row to another commit, its status follows the move. The audit also gains G16 and G17, the two C++ extraction gaps found while porting #2446 and left OWNER-EXCLUDED on 2026-10-12; Owner decision rows for OD-18 and OD-21 to OD-23; and Corrections rows for schema 10, the split, and the parts of 950686de, b1f40c57 and the route roots that moved from W7 into W6. UPSTREAM.md's Current alignment states the new v0.55.0 scope, and a 2026-10-16 entry records the split with the follow-up pull request's extraction-version boundary, 22 to 23 before its first graph-changing commit. The dated entries before it stay as written.
…s fixture The final OD-4 parity run built upstream b635dd46, indexed the mini fixture with it and compared the result with the golden through bench --graph-diff. Nodes, files and unresolved references are identical, including the this.value reference the Rust regeneration added, and every edge keeps its endpoints, kind, position, confidence and resolvedBy. The remaining differences are recorded as KEEP-RUST: upstream's refName/refKind stamp on resolved edges (kept here as resolved_refs rows), the unresolved_refs status/name_tail columns and their partial indexes, the name_segment_vocab table, and the wider idx_nodes_kind upstream's synthesis refresh uses. The regeneration log also gains the row the initialize MCP fixture lacked: the server instructions sentence about lost-event rescans, the RECOVERING banner and renames.
crates/codegraph-ui/viewer/ is `make ui` output, and `make ui-check` byte-compares it with a fresh build, so it is never hand-edited. Its minified script ends line 1 inside a string with a space, as v0.54.0's bundle did, which made `git diff --check` fail on every bundle rebuild. The `-whitespace` attribute exempts only that directory; every authored file keeps the check.
The module comment still said the upgrade leaves a schema-9 index; the test asserts CURRENT_SCHEMA_VERSION, which is 10 since resolved references moved into their own table. The comment now says the current schema.
The full-index connection set a 256 MiB page cache and a 1 GiB mmap window. The pages a full index writes and reads back then sat in the page cache (anonymous memory) and again as mapped file pages, so peak RSS counted the database twice. v0.55's database is larger (resolved_refs and more rows), so the double count grew with it. Both bulk-index entry points (Store::set_bulk_index_pragmas and the rebuild writer's set_bulk_index_pragmas) now set mmap_size to 0 and keep synchronous=OFF, the page cache, deferred WAL checkpoints and the WAL valve. Ordinary connections keep their 256 MiB window. PRAGMAs do not change what is written, so goldens and canonical graphs are unchanged. Official v0.54.0 musl against the musl build, peak RSS: this repository at 32/4/1 threads 1.156x/1.361x/1.388x -> 1.035x/1.165x/1.182x, typescript-large at 4/1 threads 1.212x/1.240x -> 0.979x/0.993x, wall unchanged.
SnapshotResolutionContext::from_store built its per-file and per-kind candidate lists by collecting each store query's Vec<Node> by value into Vec<Arc<Node>>. Rust collects that in place: the list reuses the query's buffer and keeps a node's worth of room for every 8-byte handle for the whole resolution pass (17 MiB on this repository, 18 MiB on Newtonsoft.Json; v0.54.0 has the same). The lists are now collected from a borrowing iterator, so each allocates a buffer of handles and the query result is freed at once. Elements and order are unchanged. Official v0.54.0 musl against the musl build with the full-index mmap off, peak RSS on this repository at 4/1 threads: 1.165x/1.182x -> 1.083x/1.080x.
The two depth-first loops of the same-file value references (each reader's scan and the shadowing scan) created a tree-sitter cursor for every node they popped. A cursor is a C allocation, and tree-sitter's C allocations go to the system allocator: on the musl release build that is musl's malloc, which serializes them on one lock. On redis the scans added 1.39 million allocations to 1.87 million, the summed walk stage went from 12.9 s to 35.5 s at 32 threads, and init took 25% longer than v0.54.0 for 3% more rows. Each loop now creates one cursor and resets it onto every node through named_children. Visiting order and edges are unchanged. A test counts the scan's tree-sitter allocations through a counting allocator in a child process: 1,935 for 30 readers before, at most two per reader now.
The static musl Linux builds allocate through mimalloc, whose v2 defaults advise transparent huge pages for the memory it takes from the system. On a host with THP in `madvise` mode each thread's heap then stays resident in whole 2 MiB pages: in a redis full index at 32 threads AnonHugePages was 98 MB of v0.54.0's 277 MB peak, and fd-small's peak at 32 threads was about 280 MB. The musl-only mimalloc dependency now enables the crate's `no_thp` feature (libmimalloc-sys defines MI_NO_THP), so mimalloc no longer asks for huge pages. The process and system THP settings are not touched; on an `always` host the kernel still decides, as for v0.54.0. glibc, macOS and Windows builds are unaffected, and the feature does not enter Cargo.lock. With this and the three preceding changes, official v0.54.0 musl against the musl build peaks at 0.40x-1.08x of v0.54.0's RSS on the nine other registry corpora at 32 and 4 threads (fd-small 32 threads 276 -> 118 MB); wall times stay within budget.
RustSec published three advisories on 2026-10-19 and 2026-10-20 that the Security Audit job reports for the current lock: - RUSTSEC-2026-0382, rustls-webpki 0.103.15: certificate verification may accept distrusted intermediates (fixed in 0.103.16); - RUSTSEC-2026-0381 and RUSTSEC-2026-0383, webpki-roots and webpki-root-certs 1.0.8: the bundled roots trust distrusted emSign intermediates (fixed in 1.0.10). All three reach the binary only through self_update's reqwest/rustls stack. The update is lockfile-only: three versions and checksums, no dependency added or removed. cargo audit with the current advisory database: the previous lock reports the three vulnerabilities, this one none.
Building mimalloc with `no_thp` (9ae6eb0) stopped it asking for huge pages in every command, and the short ones paid for it. The W10 sync check found single-file syncs 2.1-2.4% slower than before W10 (ts-body at 32 and 4 threads, express at 4), with two to three times the 4 KiB page faults at 4 threads (fd 2.0k -> 5.1k, express 3.2k -> 6.5k, ts-body 43.9k -> 59.2k). mimalloc asks for huge pages again, as in v0.54.0. The full index body (`init` and `index`) holds a guard that turns them off for the process with PR_SET_THP_DISABLE while it runs and back on afterwards. It reads the setting first and leaves one that was inherited, or that the kernel cannot read or change, as it was. Only the musl builds use mimalloc, so only they call it. Tests: two child-process tests read THP_enabled from /proc/self/status inside and after the guard, one starting with huge pages on and one with them inherited off.
…o characters `strip_js_comments` collected each barrel file into a `Vec<char>`, four bytes a character, before scanning it. A single-file sync of typescript-large called it 513 times, once per barrel file and worker context, and allocated 86.7 MiB doing so. It now walks bytes. Every character it acts on (the three quotes, `\`, `/`, `*` and the newline) is ASCII, and no byte of a multi-byte UTF-8 character equals one, so each removed comment starts and ends on a character boundary and the output is unchanged. That ts-body sync is about 7% faster on the musl builds at 32 and 4 threads. Tests: re_export_allocations counts the allocations of a second re-export scan of 64 KiB of TypeScript (before: 524,555 bytes for 65,527 of text, now at most twice the text), and an equivalence test compares the byte scan with the character scan on fixed inputs and on every string of up to four characters from a twelve-character alphabet.
sunerpy
changed the title
sunerpy
marked this pull request as ready for review
October 21, 2026 12:14
This was referenced Oct 21, 2026
Merged
sunerpy
added a commit
that referenced
this pull request
Oct 22, 2026
…329) Record the v0.55.0 release: #322 merged as 0796c8a, release PR #325 as 5f96a74, the push event GitHub never delivered for that merge, #327's dispatch path, Release run 39698634632, and the black-box acceptance of the downloaded binaries against v0.54.0, with the 24-cell benchmark and the older upgrade issue in default daemon mode. Close the v1.6.1 -> v1.6.2 sync: the tracked release moves to v1.6.2 with the owner's exceptions (COBOL, G16 and G17, OD-19, and the W7-W9 rows planned for the follow-up pull request). The audit's LANDED rows now name #322, and docs-check.py expects the new tracked release.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The codegraph-rs v0.55.0 program, first of two pull requests (owner decision OD-18, 2026-10-16).
It ports the portable behavior of colby
f4ddf508..b635dd46that belongs to waves W0–W6 of theplan, fixes the Rust-only defects the audit found in those areas, and records every row of the range
in
docs/upstream-sync/V1_6_2_AUDIT_2026-10-10.md(LANDEDwith its commits,KEEP-RUST,ALREADY-HAVE,N/A,OWNER-EXCLUDED, orPLANNED (follow-up PR)). Waves W7–W9 — frameworkresolvers and synthesis, the VB.NET trial,
codegraph context, Lua package loaders (#1617) and the26 performance strategies — move to the follow-up pull request, which also moves the extraction
version to 23. W10 adds the as-built docs and five memory and allocation fixes found by the final
benchmark and its sync check (owner decisions OD-24 and OD-25), so every frozen benchmark cell is
within budget and every single-file sync scenario stays within 1.02× of the build before W10.
What changes
extends/implements,Objective-C supertypes, Go embedding, Dart mixins and
implements); type-annotation referencesfor Rust, Go, Java, Kotlin, Scala, C#, Swift, Dart and PHP; instantiations; same-file value
references and static member reads; Vue, Svelte and Astro scripts folded into the component's file
node; CommonJS requires;
@flowfiles; C/C++ preParse round 2, typedefs in declarations and phantommacro functions; Kotlin, Lombok, Java enum bodies, C# accessors, CFML tags, Go aliases and docs.
name-match candidates (#2392, #2263); name matching stops for out-of-repo and namespace receivers,
receivers with no project type, test suites, out-of-scope bindings and case rules the language does
not have; per-language rules for JavaScript/TypeScript, Python, Rust, Go, C/C++, Java, Kotlin,
Scala, C#, Swift, Objective-C, PHP, Ruby, Lua, R, Dart, CFML, Solidity and Erlang; overloads by
argument fit; C# extension methods on outside types (kept beyond upstream, KEEP-RUST).
they need appears; dependents re-resolve when tsconfig paths,
package.json, C# project files,Cargo manifests or
pubspec.yamlchange; resolved references are kept (schema 10) so a syncre-resolves dependents without re-extracting them; name resolution stays linear when a repository
repeats a name many times. The sync-versus-fresh matrix runs 550 cases with none ignored.
or a sign of lost events and reports RECOVERING until the catch-up commits; an older daemon is
replaced; a cold-start session keeps the daemon it starts; MCP 2026-07-28 is answered through the
daemon proxy (OD-21) and at a too-broad root (OD-23); the prompt hook skips agent-message prompts.
page cache already holds; the resolution snapshot's per-file and per-kind lists keep only their
handles instead of the store query's node-sized buffers (17 MiB on this repository, 15 MiB on
Newtonsoft.Json); the same-file value-reference scans reuse one tree cursor instead of allocating
one per visited node, which serialized on musl's malloc lock (redis init was 25% slower than
v0.54.0 at 32 threads); a full index on the musl builds turns transparent huge pages off for its
process while it runs, so each worker's mimalloc heap no longer holds whole 2 MiB pages, and every
other command keeps them; and JavaScript re-export scans strip comments without first copying each
barrel file into characters (a typescript-large single-file sync is 7–13% faster). No graph
output changes.
repository, Flow finds its ends by exact names, lists are keyed by id, the port walk moves past a
permission-denied port, unresolved decorators and lifecycle hooks count, and non-production folders are
judged above
src/(OD-22).Index upgrade
The extraction version moves once, 21 → 22 (880ab84), and the schema goes 8 → 10 (9:
synthesis_inputs; 10:resolved_refs). A v0.54.0 index reads as outdated and is rebuilt once bycodegraph sync, a daemon's catch-up or the first MCP access;upgrade_from_v054replays a realv0.54.0 index through both paths.
Golden impact
13 new corpora (C#, Java, Swift, Objective-C, PHP, Vue, Svelte, Python bases, CommonJS, routers,
servers, mobile, synthesis), each captured at v0.54.0 behavior first so later diffs attribute to a
commit. Against the v0.54.0 goldens, rows changed in arkts, cfml, cpp, dart, erlang, go, kotlin,
mini, python, ruby, rust and typescript, and eight corpora changed only in schema; every change has
a row in the regeneration log of
docs/equivalence.md. The MCP fixtureinitialize.jsongains oneinstructions sentence. The final parity run of upstream
b635dd46on mini matches nodes, files andunresolved references; the remaining differences are recorded there as KEEP-RUST.
Verification
make pre-cion the head: 236 test suites, 6081 passed, 0 failed, 1 ignored (an existing#[ignore]large synthetic benchmark); vitest 574; committed viewer bundle byte-identical; releasearchive smoke.
scripts/regen-goldens.sh --check: 32 corpora identical.sync_convergence: 550 passed, 0ignored.
upgrade_from_v054: passed.git diff --check: clean (the generated viewer bundle isexempt from whitespace checks, c6068a2).
(50c87fe) give identical canonical graphs on the ten registry corpora and this repository; in
a single-file sync check on the musl builds every scenario stays within 1.02× of the build before
W10, and a typescript-large sync is 7–13% faster.
its last fix (c762529) unreviewed; the full-diff gate covers it.
CI Success.Performance (official v0.54.0 musl package against a musl build of this head)
codegraph initon a fresh copy,RAYON_NUM_THREADSper row, six runs per binary per cell,interleaved, the first discarded, median of the other five; a run waits for a quiet host and is
repeated when other work ran beside it. Every kept run exited 0 and produced the same canonical
graph as its binary's other runs. Xeon 6975P-C, 32 logical CPUs, Linux 7.0.0-1012-aws (THP
madvise), rustc 1.98.0, zig 0.15.1, cargo-zigbuild 0.23.0. The 24 cells are the plan's frozenset (OD-25); the budget is peak RSS at most 1.15× and init wall at most 1.10× or rows/s at least
0.90×.
14c4f93, viewer bundle excluded (rows 1.17×)typescript-large(7964e22fsrc/) (rows 1.03×)fd-small(rows 1.22×)tokio-medium(rows 1.27×)express-medium(rows 1.02×)leveldb-small(rows 1.01×)gson-medium(rows 1.24×)newtonsoft-large(rows 1.35×)alamofire-small(rows 1.46×)redis-medium(rows 1.03×)kong-medium(rows 1.00×)Owner decisions
OD-1 squash merge; OD-2 COBOL
OWNER-EXCLUDED; OD-4 mini's edges and references and the MCP fixturesfollow Rust output, checked by a final upstream parity run; OD-5 node-id changes listed in
docs/equivalence.md; OD-8 KEEP-RUSTrows; OD-14 performance overruns fixed or accepted with data; OD-18 the split; OD-19 future-schema
refusal in a later pull request; OD-21 to OD-23 the MCP 2026-07-28 and test-folder fixes; OD-24
investigate the final benchmark's RSS overruns in this pull request; OD-25 keep optimizing until
every frozen benchmark cell is within budget.
Known limits
existing gaps, not fixed (owner, 2026-10-12).
catch_up_absorbs_offline_editanddaemon_handoverhave each failed once on CI and passedon re-run.
BEGIN_COMMIT_OVERRIDE
feat(extract): record class heritage for C#, Python, PHP, Objective-C, Go and Dart
feat(extract): type-annotation references for Rust, Go, Java, Kotlin, Scala, C#, Swift, Dart and PHP
feat(extract): emit instantiations for object creation and literal construction
feat(extract): same-file value references and static member reads
feat(extract): fold Vue, Svelte and Astro scripts into the component's file node
feat(extract): CommonJS requires import their files
feat(extract): parse Flow-marked files as TSX with Flow syntax blanked
feat(extract): C and C++ preParse round 2, brace-derived scopes and types in declarations
feat(extract): Kotlin infix calls, Lombok members, Java enum bodies and C# accessor bodies
feat(extract): CFML container tags, cfinterface, cfscript-in-tag re-parse and cfquery
feat(extract): Go type aliases, defined types and standalone doc comments
feat(core): detect minified scripts by content and keep their symbols file-local
feat(resolve): a call reaches the overload its arguments fit
feat(resolve): JavaScript namespace re-exports, default aliases and destructured call results
feat(resolve): Python package re-exports, module members, pytest fixtures and Django managers
feat(resolve): C++ alias receivers, base classes by name lookup and macro-opened namespaces
feat(resolve): Dart library visibility, import URIs, parts, getters, call chains and annotations
feat(resolve): Swift extensions and conformance, Go receiver types, Rust variant paths
fix(resolve): an imports reference binds the file it names, never its own import statement
fix(resolve): never offer import statements as name-match candidates
fix(resolve): name matching stops for receivers no project type explains, test suites and out-of-scope bindings
fix(resolve): language rules for JavaScript, Python, Rust, Go, C/C++, JVM, C#, Swift, Objective-C, PHP, Ruby, Lua, R, Dart, CFML, Solidity and Erlang
fix(resolve): a C# call on an outside type reaches the project's extension method for it
fix(extract): store the sniffed language on the file row
fix(extract): Shopify JSON is Liquid only inside a theme; references stay in their theme
fix(extract): stop indexing phantom macro functions
fix(sync): retry failed imports and qualified references when what they name appears
fix(sync): re-resolve dependents when tsconfig paths, package.json, C# project files, Cargo or pubspec manifests change
fix(sync): JavaScript module surfaces and class supertypes converge with a full index
fix(watch): rescan the project when the watcher may have lost events
fix(daemon): replace a daemon left by an older install
fix(serve): keep a cold-start MCP session attached to the daemon it starts
fix(daemon): answer proxied initialize and tools/list as the direct server does
fix(mcp): serve MCP 2026-07-28 sessions at a too-broad root
fix(mcp): bound the startup sub-project scan
feat(cli): report files needing reindex and files with parse errors
fix(prompt-hook): skip a prompt that is entirely an agent message
fix(scan): honor the repository's .git/info/exclude
fix(windows): hide the console window of every git call
feat(ui): Map collapses pass-through folders and opens on the whole repo
fix(ui): directed Flow finds its ends by exact names and prefers named routes
fix(ui): key viewer lists by id
fix(ui): move past a permission-denied viewer port
fix(dead-code): unresolved decorators and lifecycle hooks count
fix(rank): judge non-production folders by the layout above src/
perf(resolve): keep name resolution linear when a repository repeats a name many times
perf(store): keep resolved references so sync re-resolves dependents without re-extracting them
perf(sync): extract the dependent files of a sync in parallel
fix(deps): update quinn-proto to 0.11.19, rustls to 0.23.46 and self_update to 1.3.1
fix(deps): update rustls-webpki to 0.103.16 and webpki-roots to 1.0.10
perf(store): stop memory-mapping the database during a full index
perf(resolve): keep only handles in the snapshot's node lists
perf(extract): reuse one tree cursor per value-reference scan
perf(cli): turn transparent huge pages off only for a full index
perf(resolve): strip JavaScript comments without copying the text into characters
END_COMMIT_OVERRIDE