Skip to content

feat(EvalTools): fail validation on problem modules no manifest reaches - #520

Merged
kim-em merged 2 commits into
mainfrom
issue-519
Aug 9, 2026
Merged

feat(EvalTools): fail validation on problem modules no manifest reaches#520
kim-em merged 2 commits into
mainfrom
issue-519

Conversation

@kim-em

@kim-em kim-em commented Aug 9, 2026

Copy link
Copy Markdown
Collaborator

This PR makes validate-manifest fail on any problem module the manifest cannot reach. It parses the header of every .lean file under LeanEval/ and rejects the ones that are neither named by some manifest module field nor imported, directly or transitively, by a module that is. manifests/problems/*.toml is CI's only entry point into the problem sources, so an unreached module is never compiled and never validated: a new problem can be added with a broken statement and CI stays green. The check parses imports rather than building anything, so it costs no build time and runs before the inventory cross-check.

Two smaller guards come with it. loadManifest now rejects non-.toml files in manifests/problems/ instead of skipping them silently, which is what let #513 land a manifest named direct_summand.lean and take its module out of the build with it; dotfiles are still ignored so a stray .DS_Store cannot wedge CI. And a manifest entry whose module names no file under LeanEval/ is now reported as a typo rather than surfacing later as an unrelated-looking unreachable-module error.

main passes the new check as-is: 245 manifest entries name 229 of the 237 modules under LeanEval/, and the other 8 are imported by modules that are.

Fixes #519.

🤖 Prepared with Claude Code

kim-em and others added 2 commits August 9, 2026 11:21
Add a coverage check to `validate-manifest`: parse the header of every
`.lean` file under `LeanEval/` and fail unless each one is named by some
manifest `module` field or imported, directly or transitively, by a module
that is. The manifest is CI's only entry point into the problem sources, so
an unreached module is never compiled and never validated.

Also reject non-`.toml` files in `manifests/problems/` instead of skipping
them silently, and reject manifest entries whose `module` names no file.

Fixes #519.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BhJsJCk5YGmsdcbz1fWT8y
Three gaps in the coverage check, from review:

Flattening a module to dot-joined text conflated distinct files.
`LeanEval/Foo/Bar.lean` and `LeanEval/Foo.Bar.lean` both read as
`LeanEval.Foo.Bar`, so a manifest entry for the nested file vouched for a
sibling Lake never builds; conversely `import LeanEval.«Foo-Bar»` did not
match `LeanEval/Foo-Bar.lean`. Key on `Lean.Name` throughout, built from
path components, and read manifest `module` fields with the shared
`parseModuleName` the inventory executable now also uses.

Skipping every dotfile in `manifests/problems/` reopened the hole it was
meant to close: the `@[eval_problem]` elaborator reads every `*.toml` there,
so a hidden `.foo.toml` could claim a tagged declaration that never reached
the catalog. Exempt `.DS_Store` alone.

Reject symlinks under `LeanEval/` rather than following them: a directory
link duplicates a subtree under a second module prefix, or points at an
ancestor and recurses forever.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BhJsJCk5YGmsdcbz1fWT8y
@kim-em

kim-em commented Aug 9, 2026

Copy link
Copy Markdown
Collaborator Author

Second-opinion review (Codex) turned up three real gaps, all now fixed in 28332a0:

  • Module identity was concatenated text. LeanEval/Foo/Bar.lean and LeanEval/Foo.Bar.lean both flattened to LeanEval.Foo.Bar, so a manifest entry for the nested file vouched for a sibling Lake never builds — the exact false negative this PR exists to close. In the other direction, import LeanEval.«Foo-Bar» failed to match LeanEval/Foo-Bar.lean and reported a false orphan. Coverage is now keyed on Lean.Name, built from path components, with manifest module fields read through a shared parseModuleName that eval_inventory also uses, so the check and the inventory cannot disagree about which file a module field names.
  • Skipping dotfiles reopened the hole. The @[eval_problem] elaborator reads every *.toml in manifests/problems/ independently of loadManifest, so a hidden .foo.toml could satisfy the attribute for a declaration that never reached the catalog. The exemption is now .DS_Store alone.
  • The walk followed directory symlinks. A link duplicates a subtree under a second module prefix, or points at an ancestor and recurses forever. Symlinks under LeanEval/ are now rejected with a clear error.

Tests grew to 13, covering each case. main still passes the check unchanged (245 manifest entries, 237 modules), and both negative cases still fail as expected — including LeanEval/EasyProblems.Extra.lean, now correctly reported as LeanEval.«EasyProblems.Extra».

@kim-em
kim-em merged commit 9e9e1b5 into main Aug 9, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI never builds problem modules that no manifest references

1 participant