Skip to content

D-CODEGRAPH-0: wide closure, merge-on-read refresh, versioned DR index (D-DYNBAG-4) - #1923

Merged
AdaWorldAPI merged 19 commits into
mainfrom
claude/clone-repositories-71a5sw
Nov 2, 2026
Merged

AdaWorldAPI merged 19 commits into
mainfrom
claude/clone-repositories-71a5sw

Conversation

@AdaWorldAPI

@AdaWorldAPI AdaWorldAPI commented Nov 2, 2026 •

Copy link
Copy Markdown
Owner

Summary

This removes slice C's 64-position limit and follows your rulings: wider call chains, presence bits for thin provisioning, sparse writes merged on read, and versioned replayable immutability kept in the DR index itself. Since the architecture review, it also makes refresh_crate atomic, closes codex's finding about binds to unmapped ids, and runs the codegraph tests in CI.

dt64 (dr)

  • MapTable::locate maps a global id to a (tile, local) pair, which closes D-DYNBAG-4's open map item.
  • LocalMap::extends checks that a map only appends ids. LocalMap::view lets a caller check a map before handing it to the table.
  • Axis::for_survivors picks the tile width from the survivor count (D-DYNBAG §4.1).
  • Versioned index. Each tile's mapping is an append-only log keyed by RouteVersion:
    • an extension logs a longer length, with no copy;
    • a replacement adds a new generation;
    • a removal logs an event.
  • MapTable and RouteLog share one sequence through a VersionClock.
  • map(frame, tile) reads the layout the frame was sealed at, and map_at(version, tile) reads any version by number.
  • A frame from another table, or a future version, is refused.
  • Lookups return a borrowed TileMap and allocate nothing.

xlog

  • closure_wide handles any number of positions. Each row stores only its non-empty 256-position blocks, with a multi-word presence mask and no tile cap. The narrow path for 64 positions or fewer is unchanged.
  • Cube::new_tiled spans several tiles.
  • Per-relation delta log, kept as a multiset: an add counts +1, a retire −1.
    • Every read sees the base plus the log, through one accessor.
    • refresh_closure recomputes only the rows that reached a retired edge's source.
    • refold runs only when called.
  • rebase accepts a new frame only if every tile extends its old map, and checks everything before it changes anything. Its private snapshot is gone; it reads the old layout through the index.
  • Cube::stage / Cube::stage_closure / Cube::commit.
    • stage checks a batch of edge operations without changing the cube or the table: the grown map must extend the current one, and every id must be placed.
    • stage_closure computes the closure as it will be after the batch.
    • commit applies the batch and cannot refuse.
  • Binds keep their global id (codex, resolved). An extending rebase or commit places a bind that was unmapped when the view was built. A mapped bind keeps its coordinate.

codegraph

  • Stores of any size now open on the multi-tile cube. Pos is u32.
  • refresh_crate is atomic: prepare, then validate, then commit.
    • Prepare and validate run against staged state:
      • the interner (SymbolStage);
      • positions and last_owner (PosStage);
      • the grown tile map, not yet given to the table;
      • the edge batch (Cube::stage);
      • the closure refresh.
    • A refusal at any point changes nothing: no interner entry, position, version, frame, log entry, lane, mask or digest.
    • The commit (table advance and seal, Cube::commit, absorbing the stages, lanes) has no error path.
    • The rest is unchanged:
      • positions never renumber;
      • growth past the reserved tiles returns NeedsReopen;
      • writes are grouped by relation and sorted ascending by (source tile, source, target).
  • Rows are interned id lanes, and an unchanged crate is skipped by its digest.
  • references compares ids instead of strings.

Bugs found by measurement and fixed

  • impact copied only the first 64 bits of each closure row, which became wrong past 64 positions.
  • callers panicked on a multi-tile store, because the cube's mask covers every reserved position while codegraph's masks cover only the symbols.
  • A refresh lost a duplicate call site. A call graph is a bag, so the fold is now a multiset.
  • Found by the review: a refused refresh could leave the graph half-updated.
    • Before the change:
      • interner entries stayed behind after NeedsReopen;
      • after a cube refusal, the positions and cube were already changed while the lanes were not;
      • a retry counted ADDs twice.
    • Fixed by the atomic refresh above.

Test Plan

  • Tests: dt64, xlog, xlog-relcomp-probe, codegraph and egglog-r2il-probe (at the egglog pin) all pass. cargo clippy --all-targets -- -D warnings and fmt are clean on each touched crate.

  • CI: rust-test.yml now runs crates/lance-graph-codegraph. It is excluded from the workspace and ran in no workflow before.

  • Atomic refresh falsifiers (graph.rs mod atomic). Each compares a full-state snapshot: interner, positions, last_owner, lanes, digests, table version, revisions and tile history, cube frame and every delta log, masks, cached closures.

    • NeedsReopen, an injected stage refusal and an injected closure refusal each leave the snapshot unchanged, and no version is issued.
    • A retry after both refusals equals the refresh that never failed.
    • m::b keeps 4 call sites, as in a clean open.
    • The snapshot does change on a successful refresh.
  • Bind and staging tests (tests/versioned_map.rs):

    • codex's example: the old view answers like a fresh one after a rebase, and also after stage/commit;
    • a mapped bind is unchanged;
    • staging issues no version and refuses a non-extending map.
  • Disable runs. Each guard added since the review fails exactly its own test:

    • absorbing before the reservation check;
    • advancing the table before staging;
    • absorbing before the closure block;
    • deduplicated keys;
    • no rebind in rebase;
    • no rebind in commit;
    • no extends check in stage.
  • Earlier falsifiers, unchanged:

    • narrow and wide closures are bit-identical up to 64 positions;
    • a 694-position, 3-tile graph matches a DFS written in the test;
    • absent blocks allocate nothing;
    • refresh equals a rebuild, with rows_recomputed == |A|;
    • refold preserves the closure;
    • rebase refuses a reorder;
    • dt64 F1–F6 and xlog F7;
    • a refreshed codegraph store matches a clean open, with one dangling call when a dependent is skipped and no renumbering.

    Every earlier guard was disable-verified. Re-pinned tests keep their old assertion as a comment.

  • Measured on mask-risc's store (695 symbols, 3 tiles), release build:

    measure result
    impact 286 ns per call
    callers 29 µs
    references 45 µs
    closures (one-time) 257 µs and 337 µs
    blocks kept RCONF 614 of 2,304; RPOSS 951 of 2,304
    refresh of an unchanged crate 1.5–1.7 ms (fails the gate)

    The refresh fails because it reads and hashes the 2.5 MB text rows file. The fix is D-CODEGRAPH-0-BIN: binary lanes with a header digest.

Decisions recorded (not implemented here)

  • P2: d-codegraph-0-binary-lanes-v1 Amendment A (§8):
    • a provenance section (charon version and commit, LLBC schema version, LLBC digest, toolchain, target and build config, source revision);
    • per-row identity (artifact, FunDeclId);
    • INVOKES_WORD to a registered DT64 reference;
    • falsifiers F9–F12.
    • Open: where "this function realises word W" comes from.
  • P3: d-dt64-method-authority-census-v1 (D-DT64-AUTH). DT64 is the one method authority.
    • Parallel identities stay as listed LAB adapters.
    • A census with a two-way ratchet: an undeclared production word fails, and so does a stale exception.
    • egglog is pinned to commit 5deaa5637a306c48c998f88d5d08792d2dbfd037.

Board

  • Architecture review entry and decisions entry.
  • Plans: d-codegraph-0-wide-closure-v1, d-dynbag-4-dr-versioned-mapping-v1, d-codegraph-0-binary-lanes-v1 (amended) and d-dt64-method-authority-census-v1.
  • STATUS_BOARD rows: D-CODEGRAPH-0-C2, D-CODEGRAPH-0-BIN, D-DT64-AUTH and D-DYNBAG-4.
  • Checks: entries_index.py --check, append_only_gate.py, the supersession index and plan_dids.py all pass.

Not to be merged automatically.

🤖 Generated with Claude Code

https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX

claude added 12 commits November 2, 2026 07:12
locate finds a global id's (tile ordinal, local) across the listed tiles and
refuses an id held by two tiles or none. extends says whether a map only
appended to an older one: prepare keeps the supplied order, so an appended id
never renumbers an existing position. Tests live in tests/dr.rs, since dr.rs
must not name its parent module.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX
…fold, refresh

closure_wide runs the same semi-naive fixpoint over any number of positions,
one row at a time; rows keep only nonzero 256-position blocks, with a
multi-word presence mask per row (no tile cap). DeltaLog appends ADD / RETIRE
edges; refold absorbs them only when called; refresh recomputes only the rows
that reached a retired edge's source and seeds additions incrementally.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX
…dt64 Axis::for_survivors

Cube::new_tiled places ids at tile_ordinal * 256 + local through
MapTable::locate (D-DYNBAG-4's global -> (tile, local) map). closure_wide
folds base + delta log and never refuses for width. add_edge / retire_edge
append to a per-relation log; refresh_closure recomputes only affected rows;
refold runs only when called. rebase adopts a new frame only if every tile's
map extends the old one: a resize is a new frame, never in place (D-DYNBAG
§4.3). Axis::for_survivors is §4.1's width rule. The narrow single-tile path
is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX
…ds whole rows

Pos is u32; the tile layout comes from dt64's Axis::for_survivors, one tile
keeps the single-tile path bit for bit, more tiles use Cube::new_tiled.
Closures are WideRows from closure_wide. impact copied only the first word of
a closure row, which was wrong once a graph has more than 64 positions; it now
reads the whole row. TooMany and ClosureWidth are gone; their two tests are
re-pinned to the new truth (a 70-function chain closes across words; 300
symbols open across two tiles).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX
…bol width

xlog tests/wide_closure.rs: narrow vs wide bit-identical up to 64 positions;
a 694-position three-tile graph with cross-tile cycles against a DFS written
in the test; absent blocks allocate nothing (with its anti-vacuity twin);
refresh equals a rebuild after every step, with rows_recomputed == |A|;
refold is explicit and preserves the closure; rebase accepts appends and
refuses a reorder; growth is a new frame.

codegraph: on a multi-tile store the cube's support mask covers every
reserved position (12 words for 694 symbols) and callers panicked on the
width mismatch; partners now trims to the symbol width. Found by running on
mask-risc's real store; queries_work_across_tiles pins it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX
One accessor returns a relation's effective lanes: the base when its log is
empty, otherwise a cached fold of base and log, dropped on every add_edge,
retire_edge, refold and rebase. Terminals (support, bag_count), the narrow
closure, closure_wide and materialize_rows read through it, so views no
longer see stale edges until an explicit refold. Reads never modify the base
or the log; refold stays the only thing that replaces the base.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX
…g is a multiset

Codegraph::refresh_crate re-reads one crate's rows and updates the resident
graph without rebuilding: rows carry their crate, positions never renumber,
new symbols extend the last tile under a new frame (growth past the
reservation is NeedsReopen), edge changes go to the cube's delta log grouped
by relation and in ascending (src tile, src, dst) order, cached closures are
refreshed. refold() absorbs the logs only when called.

A call graph is a bag: two call sites of one pair are two edges. The delta
log fold counts occurrences (ADD +1, RETIRE -1, floored at 0) and the refresh
diffs multisets; the closure sees an edge only while its count is above 0.
Found because a refresh to rev2 counted one call site of area_of where a
clean open counts two.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX
Measured on mask-risc's store (17,155 rows), release: refresh_crate of an
unchanged crate 24,589-36,259 us, of which reading the rows file is
14,078-19,708 us. A millisecond result is a failed gate: the reader builds
one String per field (about 120,000 allocations) and refresh re-interns and
diffs through string-keyed maps. Recorded here before the fix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX
…sed references

Rows are stored as u32 ids into one interner plus a predicate code; parsing
allocates only for names not seen before. refresh_crate skips a crate whose
rows file digest is unchanged (no delta) and diffs changed crates with sorted
u64 keys. references compares ids, not strings.

Measured on mask-risc's store (17,155 rows), release: reading the rows file
14-20 ms -> 6.8-10.3 ms; refresh of an unchanged crate 25-36 ms -> 1.5-1.7 ms;
references 122 us -> 45 us; impact 286 ns. Still a failed gate: the remaining
cost is reading and hashing the 2.5 MB text file, which only a format change
on the ruff_charon side removes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX
…h it

Operator ruling: versioned, replayable immutability is the DR global -> local
index itself; reading an older version is the same lookup with the version
as part of the key.

dt64::dr: each tile's mapping is an append-only log keyed by RouteVersion. An
extension logs a longer length (no copy), a replacement adds a generation, a
removal logs an event; nothing is updated in place. MapTable and RouteLog
share one RouteVersion sequence through a VersionClock. map(frame, tile)
reads the layout the frame was sealed at; map_at(version, tile) reads any
version by number; a frame from another table or a future version is refused.
Reads return a borrowed TileMap that respects the version's length and
allocate nothing.

xlog: Cube::rebase drops its private per-tile snapshot and reads the old
layout through the index; a cube keeps answering after the table advances.

Re-pinned to the new rule (old assertion kept as a comment): dt64
tests/dr.rs (two), xlog tests/cube.rs, xlog-relcomp-probe
an_old_map_frame_replays_its_own_layout (method admission is still gated by
the bind store's route versions, not the tile map).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX
…AG-4), binary lanes

Plans: d-codegraph-0-wide-closure-v1, d-dynbag-4-dr-versioned-mapping-v1,
d-codegraph-0-binary-lanes-v1. STATUS_BOARD: D-DYNBAG-4 in PR (partial),
D-CODEGRAPH-0-C2 in PR, D-CODEGRAPH-0-BIN queued. Gap-map entry: the wave's
rulings, results, measured numbers, the failed refresh gate and what fixes it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX
@coderabbitai

coderabbitai Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 1e339545-0d52-49d5-9134-dc6e0e07f740
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-02T08:51:09.185216Z dbe4505 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: dbe4505767

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread crates/lance-graph-xlog/src/cube.rs Resolved
…h next-step proposal)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX
…e::stage/commit, binds keep their global id

refresh_crate now stages the interner (SymbolStage), the positions
(PosStage), the grown tile map (not handed to the table), the edge batch
(Cube::stage, checked without changing the cube) and the closure refresh
(Cube::stage_closure). A refusal at any of these changes nothing: no
interner entry, position, last_owner, table version, frame, log entry,
lane, mask or digest. The commit (table advance + seal, Cube::commit,
absorb, closures, lanes) has no error path.

Cube::rebase checks before it changes anything. Node::Bind keeps the
global id it binds, and an extending rebase or commit places a value that
was unmapped when the view was built (codex P2 on #1923).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX
…2 BIN amendment, P3 DT64 authority census); pin egglog

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX
…ow that cited the floating line

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX
…ble, smallest adapter, falsifiers; no code)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX
@AdaWorldAPI
AdaWorldAPI merged commit 2e19a5a into main Nov 2, 2026
16 checks passed
AdaWorldAPI pushed a commit that referenced this pull request Nov 2, 2026
…its merge point

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX
AdaWorldAPI added a commit that referenced this pull request Nov 2, 2026
…1a5sw

D-TGL: F-LAND / F-DISCARD lab, Thought-Git lifecycle contract, post-merge record for #1923
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants