Repository navigation
feat: port physics-intern loop-discipline patterns (verifier, reviser, saturation, surveyor) - #9
Conversation
Idea ported from huggingface/physics-intern reviewer prompt: require the
verifier to enumerate every check it ran with type=constraint|conjecture
and outcome=PASS|FAIL|N/A. Converts opaque CORRECT verdicts into an
auditable record of coverage.
- `correct` requires >=3 constraint checks PASS, 0 constraint FAIL
- empty CHECKS PERFORMED block forces CONFIDENCE < 0.30
- skill verifier one-liner now reports `CHECKS: {n_pass}/{n_total}`
Touches both math and physics, both skill prompts and library prompts.
Static prompt change only — parser-side enforcement of the >=3 floor
will land in a follow-up.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Idea ported from huggingface/physics-intern strategy-reviser prompt: the reviser must triage every issue from the critique with one of three verdicts before deciding patch-vs-restart. Resolves two failure modes at once — nitpick chasing (now `decline`) and silent ignoring of valid concerns (now an explicit `decline` rationale appears in the changelog). - `accept` — issue is real, will change the solution - `decline` — real concern, low marginal value (no change) - `dismiss` — critique is wrong, counter-argument required Whole-critique decline+dismiss is explicitly a legitimate outcome (returning the previous solution verbatim). Orchestrator-side handling of the all-declined case (re-generate vs. budget accounting) lands in a follow-up. Touches both math and physics, both skill prompts and library prompts. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Idea ported from huggingface/physics-intern senior-critic prompt: detect loop saturation by tracking the running history of error_taxonomy categories per iteration, and inject a verifier-side prompt addendum when any category has fired >=2 times. Tells the verifier to either confirm resolution, escalate to a structural objection, or recommend termination-as-best-effort — but not to file another routine instance of the same category. Decoupling preserved: ONLY category labels and counts cross from prior iterations into the verifier. Never critique text, never solution text, never generator reasoning. - RunState gains `critique_category_history: list[tuple[int, str]]` - OracleRouter gains `saturation_signal()` and a saturation_addendum_fn factory parameter (None default — backwards compatible) - build_verifier_extra_system composes saturation block with existing adversarial + atom-focus blocks via nested combine() - MathAgent + PhysicsAgent expose _saturation_addendum() returning the domain-specific constant - agent.py records (iteration, error_cat) immediately after classify_errors at line 923 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Idea ported from huggingface/physics-intern background-surveyor agent. One Claude call before the GVR loop, given ONLY the problem statement, produces a structured list of: KNOWN_PITFALLS (2-6, problem-specific) CANONICAL_METHODS (1-4) SANITY_CHECK_CANDIDATES (3-6, typed constraint|conjecture) The pitfall list is appended to the generator's system prompt (to avoid) and the sanity-check candidates are appended to the verifier's (to seed the CHECKS PERFORMED block from patch #1). Decoupling preserved: the surveyor never sees any solution, runs exactly once, output is advisory. - New module src/alethic/surveyor.py (parser + formatter + survey call) - New tests tests/test_surveyor.py (parser + formatter, 6 tests) - AgentConfig.enable_surveyor: bool = False (off by default) enabled in thorough + extreme presets - MathAgent + PhysicsAgent expose _survey_guidance(role) for the role-specific suffix appended after the data block - solve() invokes survey() once after system-prompt build, then inlines the formatted block into generator_system + verifier_system - Surveyor failure (API error, parse error) degrades to empty result with a warning; the loop continues without scaffolding Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cross-model smoke test (2026-05-28)Ran the patches against three non-Claude model families via NVIDIA NIM, routed through the existing Methodology
Patch fire-check
Notable findings
RecommendationMerge as-is. The PR does not regress non-Claude users — Kimi still produces useful verifications, just unstructured. The patches' value proposition (structured output for parseability) is realized fully on Claude and partially on Nemotron. Open follow-up: the Kimi finding strengthens the case for the deferred Saturation (patch #3) and ISSUE TRIAGE on non-Nemotron models remain unverified; both can be addressed by follow-up testing without blocking this PR. Smoke-test artifacts (not committed)
|
Patch #3 from this branch shipped without unit coverage; smoke testing on NVIDIA NIM could not reliably force the same critique category to recur, so the wiring was unverified end-to-end. These 9 tests exercise it deterministically: - saturation_signal() category counting from RunState history - _build_saturation_block() threshold gating (SATURATION_TRIGGER_COUNT=2) - Backwards-compat path (no saturation_addendum_fn -> no block) - Top-category selection from mixed history - route() composes the block into RoutingDecision.verifier_extra_system - Real PHYSICS_SATURATION_AWARENESS_ADDENDUM template substitution - Negative case: route() does NOT inject below threshold Test count: 1374 -> 1383 (+9). All 1383 pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Follow-up: saturation test coverage added (
|
Patch #1's prompt rule says an empty or insufficient CHECKS PERFORMED block must cap CONFIDENCE below 0.30. Cross-model smoke testing (via NVIDIA NIM) showed prompt-only enforcement degrades silently against non-Claude models — Kimi K2.6 was observed emitting prose Markdown headers (e.g. "## Dimensional Analysis Check") instead of the required [name | type=X | outcome=Y] rows, while still self-reporting high confidence. The empty-block floor rule never fired because the block had been *replaced* with prose, not omitted. This commit closes that gap with parser-side enforcement: - New _parse_checks_performed(text) parses the block and counts type=constraint outcome=PASS/FAIL entries. - New _apply_checks_floor(result, text) caps confidence at 0.30 when constraint-PASS count < 3. - verify() applies the floor when config.enforce_checks_floor is True. - AgentConfig.enforce_checks_floor: bool = False (back-compat for any caller using a bare AgentConfig — existing tests unaffected). - All four presets (quick/default/thorough/extreme) opt in to True. 22 new tests in tests/test_checks_floor.py cover: - parsing edge cases (bold markdown wrapper from 1b6f377, conjecture vs constraint distinction, FAIL outcomes, N/A outcomes, block terminated by next section heading) - floor application (no-op when satisfied, floor when missing or insufficient, no inflation of already-low confidence, preserves verdict/critique/issues fields) - verify() integration (back-compat with bare AgentConfig, floor applied when flag enabled, all presets enable the floor) Test count: 1383 -> 1405 (+22). All 1405 pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Follow-up: parser-side CHECKS PERFORMED floor (
|
| Patch | Status |
|---|---|
#1 CHECKS PERFORMED (d93c076) |
Verified on Claude + Nemotron; parser-side enforcement now backstops non-compliant models |
#2 ISSUE TRIAGE (a821646) |
Verified on Claude + Nemotron |
#3 Saturation (98059ac) |
Verified by unit tests (1076bb4) |
#4 Surveyor (a8ea1b6) |
Verified on Claude + Nemotron + DeepSeek + Kimi |
Parser-side floor (692a6d3) |
Closes original PR open question #1 |
Patch #2 (PR #9) allows the reviser to mark each verifier issue as accept/decline/dismiss. When all issues are declined or dismissed, the returned "revision" is functionally a no-op (prior solution verbatim). The orchestrator does NOT special-case this — the verifier remains the sole authoritative gatekeeper (decoupled-verification invariant). But the pattern is now observable via a new structured event so future analysis can detect verifier↔reviser disagreement loops. Changes: - Solution.triage_summary: dict[str, int] | None — counts accept/decline/ dismiss verdicts, None for generator-produced Solutions - _parse_triage_verdicts(text) helper parses the ISSUE TRIAGE block - revise() populates triage_summary from the reviser's response - EventType.REVISER_ALL_DECLINED + emit in agent.py revision sub-loop when accept count == 0 and other verdicts > 0 - Tolerates bold markdown wrapper (**ISSUE TRIAGE**) per 1b6f377 fix 17 new tests in tests/test_triage_observability.py: - ISSUE TRIAGE parser edge cases (all-accept, all-decline, mixed, bold wrapper, block termination) - revise() populates triage_summary correctly (None when block absent) - all_declined predicate covers boundary cases (empty dict, None, any-accept-means-false) - EventType reachability Test count: 1405 -> 1422 (+17). All pass. Closes the last open question on PR #9 (all_declined orchestrator handling): keep current behavior, add structured observability. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Follow-up: all_declined observability (
|
| Patch / commit | Status |
|---|---|
#1 CHECKS PERFORMED (d93c076) + parser-side floor (692a6d3) |
✅ Verified on Claude + Nemotron; parser backstop for non-compliant models |
#2 ISSUE TRIAGE (a821646) + all_declined event (dff14da) |
✅ Verified on Claude + Nemotron; no-op revisions now observable |
#3 Saturation (98059ac) + tests (1076bb4) |
✅ Wiring proven by 9 deterministic tests |
#4 Surveyor (a8ea1b6) |
✅ Verified on Claude + Nemotron + DeepSeek + Kimi |
All open questions from the original PR description are resolved. PR ready for review.
The repo had no CI of any kind — no .github/, no pre-commit — while carrying four open PRs totalling ~10,400 unreviewed added lines. Every merge to date has been gated only by someone running the suite locally. Gates on pytest and mypy. Ruff runs advisory (continue-on-error) because main carries 111 pre-existing findings, 93 of them in tests/ and mostly unused imports and import ordering; gating on that today would block every merge on unrelated debt. Clean those in a dedicated pass, then drop continue-on-error. No ANTHROPIC_API_KEY is provided, deliberately: the suite is fully mocked and must stay that way, so a test that starts needing a live key will fail here. Landing this via PR #12 rather than a separate PR so it takes effect after PR #11 brings mypy to 0, which means it gates the two merges that actually carry risk (#10 at 8467 lines, #9 with three conflict hunks). Assisted-by: Claude Code:claude-opus-5 Signed-off-by: A. Friedrich <alex@f2media.de>
Brings in PR #11 (mypy), PR #12 (metric split, evidence, CI), PR #10 (v3.8 tree search). Three conflicts, all keep-both: models.py EventType REVISER_ALL_DECLINED (#9) + the 5 v3.8 search events -> 16 members (10 in v3.7, +5, +1) models.py AgentConfig enable_surveyor/enforce_checks_floor (#9) + search_mode/search (#10) test_new_types.py len(EventType) assertion 11 vs 15 -> 16 Also fixes a mypy regression this merge exposed: _parse_survey() bound `m` twice for two different types — Match[str] from enumerate() at line 115, then Match[str] | None at 133. Renamed the second to `sanity` rather than annotating it, since the reuse was the actual defect. Without this, merging #9 would have silently undone part of #11's mypy cleanup. Verified: 1707 passed, 2 skipped, 3 xfailed; mypy 0 errors across 41 files. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: A. Friedrich <alex@f2media.de>
No PR owned the version bump, so the repo declared 3.7.0 while main already contained the v3.8 hierarchical proof search milestone. Bumps all five sites: pyproject.toml, .claude-plugin/plugin.json, .claude-plugin/marketplace.json, src/alethic/__init__.py, and the CLAUDE.md manual-install DEST path. 3.8.0 rather than 3.7.3: #10 and #9 both add features (tree search; surveyor and checks-floor). Tree search is opt-in — search_mode defaults to "flat" in all four presets, verified — so the classic GVR path is unchanged. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: A. Friedrich <alex@f2media.de>
Nine scripts that had accumulated untracked. Committing them before they go
the way the calibration data nearly did — they sat on one machine for months.
The two that matter for what comes next:
baseline_raw.py raw model performance with NO Alethic harness — the
control arm any "does the GVR loop help?" claim needs
judge_with_claude.py batches those baselines to `claude -p` for independent
ground-truth judging, writing claude-verdicts.jsonl
Together they are most of the instrument the RQGM analysis calls rec #1 (the
harvested (problem, solution, true_verdict) labelled set), which is the only
design that can reach correlated verifier blind spots. Neither has ever been
run: no baseline*.jsonl or claude-verdicts* exists anywhere in the repo.
The rest are supporting probes: baseline-{flash,nano,qwen,super}.sh driver
scripts, probe_model.py / probe_patches.py, and smoke_test_pr9.py (exercises
the four PR #9 patches end-to-end against a non-Claude model via NVIDIA_API_KEY
— it needs a live key, so pytest does not cover it).
Tracked as preserved, not as blessed: unrun, unreviewed, and not wired into CI.
Assisted-by: Claude Code:claude-opus-5
Signed-off-by: A. Friedrich <alex@f2media.de>
Summary
Four small, independently revertable patches inspired by huggingface/physics-intern's multi-agent prompts. Each commit is self-consistent and the test suite passes at every step.
CHECKS PERFORMEDblock (d93c076): mandatory per-verification checklist with[name | type=constraint|conjecture | outcome=PASS|FAIL|N/A]rows. Advisory rule:correctrequires ≥3constraintPASS and 0 FAIL; empty block ⇒ confidence floored at 0.30. Mirrored in both library prompts and skill references.ISSUE TRIAGEblock (a821646): per-issueaccept/decline/dismissverdicts. All-decline+all-dismiss is explicitly a legitimate no-op revision (returns prior solution verbatim).98059ac):RunState.critique_category_historyrecords(iteration, error_category)after each verify. When the same category recurs ≥2 times,OracleRouter.saturation_signal()triggers; the verifier'sextra_systemgetsSATURATION_AWARENESS_ADDENDUM. Decoupling preserved — only category labels + counts cross iterations, never critique text.a8ea1b6): newsrc/alethic/surveyor.py. One Sonnet call before any generation, output seeds both generator and verifier system prompts with domain-specific pitfalls.enable_surveyor=Trueinthoroughandextremepresets only (default off for back-compat). Survey failure degrades to empty result + warning; loop continues.Net diff: +641 / -27 LOC across 12 files (8 source, 4 skill references), plus 1 new module + 1 new test file.
Architecture notes
extra_system=/ system-prompt-appending channels. Keeps the static prompts state-shape-free.build_verifier_extra_system()to composecombine(combine(adversarial, directive), saturation). Thesaturation_addendum_fnconstructor param is optional (defaultNone), preserving back-compat for callers that don't supply one.Test plan
survey()needed becauseenable_surveyordefaults toFalse.4k tokens per session on Opus ($0.05). Benchmark against `eval/harness.py math-sample.json` before considering default-on for the `default` preset.Deferred / follow-ups
n_constraint_pass ≥ 3enforcement for Patch Claude/configurable agent settings w iybi #1. Currently advisory in the prompt; ~30 LOC insubagents.py:_parse_verificationwould make it a hard gate. Left out of this PR to keep diff small and to first observe whether Opus respects the advisory floor.all_declinedorchestrator handling for Patch Update CLAUDE.md with preset documentation and configurable threshold #2. A reviser that accepts/declines/dismisses everything produces a no-op revision; this PR lets the orchestrator treat it like any other revision (counts against budget). Re-generate-from-scratch behavior is a separate decision.