Skip to content

ruff_charon: binary rows format with provenance and per-row FunDeclId (D-CODEGRAPH-0-BIN, PR A) - #133

Merged
AdaWorldAPI merged 2 commits into
mainfrom
claude/codegraph-bin-writer
Nov 2, 2026
Merged

AdaWorldAPI merged 2 commits into
mainfrom
claude/codegraph-bin-writer

Conversation

@AdaWorldAPI

Copy link
Copy Markdown
Owner

Summary

ruff_charon now stores each crate's relation rows as <crate>.rows.bin as well as rows.tsv. Ids are held in fixed-width lanes, labels live in their own sections, and a header digest covers everything after it. The lance-graph codegraph reader (PR B, a separate PR) will be able to skip an unchanged crate by reading 64 bytes and decode a changed crate without parsing text. The format follows lance-graph plan d-codegraph-0-binary-lanes-v1.md §2 and Amendment A §8.

store_bin.rs contains a pure writer and a verifying reader.

  • Header: 64 bytes. It holds a magic, major/minor version, a PARTIAL flag, a digest over [32..file_len), file_len, the counts, and reserved bytes that must be zero.
  • Directory: an aligned, ascending section directory. Every current tag is marked must-understand (bit 31). Unknown tags without that bit are skipped.
  • Codebook lanes: predicate, grade and file are codec9 lanes, using ndarray EpiLane9 codeword semantics with presence in bit 8. Codeword 0 means "absent" and never decodes as an entry. A crate with more than 511 files switches the file lane to a raw u32 lane.
  • Id lanes: u32 lanes for subject, object, MAY-dispatch target, start and end. MAYT is precomputed exactly as the lance-graph reader's dispatch_target derives it.
  • Symbol table: first-seen order, with SYM_HASH (the digest of each label) and SYM_OWNER (the crate-prefix id).
  • FUNDECL lane: each row's Charon FunDeclId. With one LLBC artifact per file, ARTREF is omitted.
  • OBJ_KIND lane: today every object is a symbol. WORD and METHOD are reserved for INVOKES_WORD, whose source association is still an open decision.
  • PROV section: one record per LLBC artifact. It holds the Charon version, the Charon commit, the LLBC schema version, the LLBC digest, the rustc commit, the target triples plus Charon options, and the source revision. A field is UNKNOWN when it is not known, and the reader keeps UNKNOWN distinct from every value, including the string "UNKNOWN".
  • Reader refusals: each malformed shape has its own ReadError variant.

Store:

  • import writes rows.bin first, then rows.tsv, residuals, and files last.
  • The TSV export stays on by default until the lance-graph reader reads the binary format; with_tsv_export(false) turns it off.
  • rows() reads rows.bin, falling back to rows.tsv for a crate stored before this change; rows_tsv() keeps the text path.
  • import_with(.., &SourceProv) takes the rustc commit and source revision from the caller. The store never guesses them.
  • provenance() reads the record back.
  • build.rs captures the Charon checkout's commit; any failure leaves the field UNKNOWN and never breaks the build.

Llbc and Relation: Llbc gains charon_version and llbc_digest. Relation gains fun: Option<FunDeclId>, set for every row harvested from a function.

Golden fixture: tests/data/golden/cg_lib.rows.bin and .rows.tsv pin the bytes for the lance-graph reader. Re-bless them with CHARON_BLESS=1. Their provenance record leaves the build-dependent fields UNKNOWN.

Test Plan

  • cargo test in crates/ruff_charon: every suite passes.
    • store_bin: 27 tests.
    • store_bin_store: 5 tests.
    • Existing suites: 3, 8, 8 and 10 tests.
  • cargo clippy --all-targets -- -D warnings and cargo fmt are clean.
  • Digest golden vectors: three vectors, computed by an independent port of the reader's fold rather than by this code.
  • Disable checks: each disable was applied with an asserted anchor, run, and reverted; each turned exactly its intended test red.
    • The reader's digest check: the flipped-byte test fails.
    • Accepting an unknown must-understand tag: its refusal test fails.
    • Clamping an out-of-range symbol id: its refusal test fails.
    • Swapping subject and object in the written rows: the row-for-row bin/TSV comparison and the golden test fail.
    • Dropping FUNDECL: the per-row FunDeclId test fails.
    • Ignoring the caller's source revision: the provenance test fails.
  • Existing-test change: one test, readers_refuse_unknown_versions_and_missing_columns, now calls rows_tsv(), because it exercises the TSV reader specifically. trait_impl_calls_are_not_dangling writes a TSV-only store by hand and passes unchanged through the fallback.

Limits:

  • The anti-vacuity check for a row with no site is satisfied by the CRATE row in every fixture; no fixture has a function row without a site.
  • load reads the LLBC file twice, once for the digest and once through Charon's file deserializer. Charon has no JSON from-bytes API without a new dependency.

Generated by Claude Code

…RAPH-0-BIN)

store_bin.rs: `<crate>.rows.bin` per lance-graph plan d-codegraph-0-binary-lanes
§2/§8 - 64-byte header (magic, major/minor, PARTIAL flag, digest over
[32..file_len), file_len, counts), an aligned section directory, codec9
codebook lanes (PRED, GRADE, FILE <=511 else u32, OBJ_KIND), u32 id lanes
(SUBJ, OBJ, MAYT, START, END, FUNDECL), a symbol table with SYM_HASH and
SYM_OWNER, and a PROV section where UNKNOWN is distinct from every value.
The reader refuses each malformed shape with its own error. Not yet wired
into Store.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX
…e and FunDeclId

`Store::import` writes `<crate>.rows.bin` first, then the TSV (on by default
until the lance-graph reader reads the binary format), residuals and files.
`rows()` reads the binary file and falls back to `rows.tsv` for a crate stored
before the format existed; `rows_tsv()` keeps the text path. Each file carries
one provenance record: Charon version, the Charon commit captured at build
time, the LLBC digest, target triples and options, and rustc commit and source
revision when the caller supplies them (`import_with`); anything not known is
UNKNOWN. Every relation row carries the FunDeclId of the function it came
from. A golden `cg_lib.rows.bin` pins the bytes for the lance-graph reader.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X6y3drwKSE2zSgoexheLFX
@coderabbitai

coderabbitai Bot commented Nov 2, 2026

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 7460b61b-87cb-4fcb-ba1e-5c9566c4ce9d
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-02T16:21:49.239839Z 63c7471 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@AdaWorldAPI
AdaWorldAPI merged commit 6c3d1bf into main Nov 2, 2026
36 of 37 checks passed

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 63c74712fe

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread crates/ruff_charon/src/store.rs Resolved
Comment thread crates/ruff_charon/src/store_bin.rs Resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants