Expand eval corpus to 20 cases with stratified stats

README.md

@@ -21,6 +21,8 @@ does bounded, mechanical generation on a private GPU host running
   the workstation.
 - **[docs/cloud-orchestrator-results-2026-09-07.md](docs/cloud-orchestrator-results-2026-09-07.md)**
   — dated Cloud Agent run of the stub corpus and apply gate (no GPU).
+- **[docs/cloud-corpus-stats-2026-09-07.md](docs/cloud-corpus-stats-2026-09-07.md)**
+  — dated 20-case stub harness with `by_tool` / `by_category` rates (no GPU).
 - **[docs/security-scan-results-2026-09-06.md](docs/security-scan-results-2026-09-06.md)**
   — dated Gitleaks, GitHub alert, tracked-tree, and deployment-safety results.
 - **[examples/](examples/)** — public-safe client config templates. Copy them

docs/cloud-corpus-stats-2026-09-07.md

@@ -0,0 +1,67 @@
+# Cloud Agent 20-case corpus stats — 2026-09-07
+
+Dated note from a Cursor Cloud Agent. This VM has **no GPU and no Ollama**.
+The worker is the loopback stub. These rates measure the scorer, stub
+failover policy, and stdio MCP loop. They are **not** live
+`qwen3.5:9b` / `devstral-small-2` quality.
+
+Commands (all passed):
+
+```bash
+PYTHONPATH=src .venv/bin/python -m unittest discover -s tests -v
+.venv/bin/python scripts/run_eval.py
+.venv/bin/python scripts/run_orchestration.py
+.venv/bin/python scripts/run_harness.py --backend stub --profile golden --fast-ms 1 --strong-ms 1
+.venv/bin/python scripts/run_harness.py --backend stub --profile observed --fast-ms 1 --strong-ms 1 --out eval-runs/observed-20
+.venv/bin/python scripts/run_harness.py --backend stub --profile golden --orchestrate --fast-ms 1 --strong-ms 1
+```
+
+## Unit suite
+
+**109 tests passed.** Fixture scoring now covers 20 cases / 41 fixtures.
+`by_tool` and `by_category` rates are tested without MCP.
+
+## Stub MCP harness (not GPU time)
+
+Golden, 20 jobs, 20 attempts: **pass@1 = 1.00**, escalated = 0.00.
+
+Observed, 20 jobs, 42 attempts: **pass@1 = 0.00**, **pass@end = 1.00**,
+escalated = 0.40. First failures: format 6, structure 10, behavior 6.
+
+| Tool | n | pass@1 | pass@end | escalated | first_failure |
+| --- | ---: | ---: | ---: | ---: | --- |
+| `local_refactor` | 10 | 0.00 | 1.00 | 0.20 | format 6, structure 6 |
+| `local_code` | 3 | 0.00 | 1.00 | 1.00 | behavior 3 |
+| `local_generate_tests` | 3 | 0.00 | 1.00 | 1.00 | behavior 3 |
+| `local_explain` | 2 | 0.00 | 1.00 | 0.00 | structure 2 |
+| `local_review` | 2 | 0.00 | 1.00 | 0.00 | structure 2 |
+
+| Category | n | pass@1 | pass@end | escalated | first_failure |
+| --- | ---: | ---: | ---: | ---: | --- |
+| extract | 4 | 0.00 | 1.00 | 0.00 | format 2, structure 2 |
+| prompt_contract | 2 | 0.00 | 1.00 | 1.00 | structure 4 |
+| rename | 2 | 0.00 | 1.00 | 0.00 | format 2 |
+| split | 2 | 0.00 | 1.00 | 0.00 | format 2 |
+| implement | 3 | 0.00 | 1.00 | 1.00 | behavior 3 |
+| tests | 3 | 0.00 | 1.00 | 1.00 | behavior 3 |
+| explain | 2 | 0.00 | 1.00 | 0.00 | structure 2 |
+| review | 2 | 0.00 | 1.00 | 0.00 | structure 2 |
+
+Vague-prompt cases (`whitespace_extract_vague`, `extract_dataclass_vague`)
+stay wrong on every fast call, so they escalate to strong (pass@3).
+Implement and test cases fail behavior on the first fast reply and
+escalate. Format/structure-only cases repair on fast (pass@2).
+
+Categories label the committed corpus. They are **not** a learned
+task classifier.
+
+## Apply gate
+
+Unchanged 13-job scripted loop: 9 delegated, 11 applied, 2 held.
+Keep jobs did not call `local_*`.
+
+## What this note is not
+
+- Live tok/s or pass@1 on the workstation tags
+- Proof that desktop Cursor's tool picker chose `local_refactor`
+- A claim that stub `mcp_ms` is model latency

docs/evaluation-protocol.md

@@ -63,6 +63,23 @@ The same scorer runs against:
 | `implement_clamp` | `local_code` | Implement `clamp` from a spec, no starter file |
 | `explain_clamp` | `local_explain` | Prose: names the function and bounds; mentions clipping |
 | `review_login` | `local_review` | Prose first-pass: flags None and missing auth. **Not** an apply |
+| `extract_dataclass` | `local_refactor` | Move `Person` + `as_person` into `person_model.py`; `tagged()` oracle |
+| `extract_dataclass_vague` | `local_refactor` | **Same checker**, vaguer prompt (`Move the person type…`) |
+| `rename_three_files` | `local_refactor` | `fetch` → `load` across `http_client.py` / `service.py` / `main.py` |
+| `split_settings` | `local_refactor` | Move retries/timeout constants into `settings.py` |
+| `implement_slug` | `local_code` | `slugify('Hello World') == 'hello-world'` |
+| `implement_median` | `local_code` | Odd-length `median([1, 3, 2]) == 2` |
+| `test_clamp_execute` | `local_generate_tests` | Generated clamp tests are imported and executed |
+| `test_median_execute` | `local_generate_tests` | Generated median tests are imported and executed |
+| `explain_mean` | `local_explain` | Prose: names `mean` and that it divides the sum |
+| `review_divzero` | `local_review` | Prose first-pass: flags a missing zero-denominator check |
+
+That is **20 cases**. Harness JSON (`scripts/run_harness.py --out`) adds
+`by_tool` and `by_category` rates (`pass@1`, `pass@end`, `escalated`,
+`first_failure`) so a paper can stratify without dumping transcripts.
+Categories (`extract`, `prompt_contract`, `rename`, `split`, `implement`,
+`tests`, `explain`, `review`) are labels on the committed corpus, **not**
+a task classifier.
 
 Known-fail fixtures are part of the corpus. They prove the scorer can
 tell layers apart:
@@ -196,8 +213,10 @@ live script. It now uses the shared fence extractor. Prefer
 After repeated live runs, a paper may claim:
 
 - Layer-conditional rates on this corpus (format vs structure vs behavior).
+- Tool- and category-stratified `pass@1` / `pass@end` on the 20-case corpus.
 - That a vaguer prompt raises structure failures on the same oracle
-  (`whitespace_extract` vs `whitespace_extract_vague`).
+  (`whitespace_extract` vs `whitespace_extract_vague`; `extract_dataclass` vs
+  `extract_dataclass_vague`).
 - That shape-only test generation overstates success relative to
   executed tests (`test_add_execute`; A6 now uses this checker).
 - That keep-vs-delegate and accept/rewrite/reject are enforceable as a

docs/phase3-log.md

@@ -8,6 +8,8 @@ Reproducible commands, complete 2026-09-06 observations, and limitations:
 Scoring method: [evaluation-protocol.md](evaluation-protocol.md).
 Cloud-safe stub + apply-gate run:
 [cloud-orchestrator-results-2026-09-07.md](cloud-orchestrator-results-2026-09-07.md).
+20-case corpus + stratified stub rates:
+[cloud-corpus-stats-2026-09-07.md](cloud-corpus-stats-2026-09-07.md).
 
 | Date | Tool | Model | Task | Result | Notes |
 | --- | --- | --- | --- | --- | --- |
@@ -16,6 +18,7 @@ Cloud-safe stub + apply-gate run:
 | 2026-09-06 | local_refactor | fast | extract module-level whitespace helper | accepted | Real Ollama; generated module parsed and preserved behavior across 3 executed cases; warm run 6.7s |
 | 2026-09-06 | local_refactor | strong | extract module-level whitespace helper | accepted | Real Ollama; generated module parsed and preserved behavior across 3 executed cases; warm run 25.6s; model reported 46% CPU / 54% GPU at 16K context |
 | 2026-09-07 | all local_* eval tools | stub | 10-case corpus + apply gate | accepted / rewritten / rejected per job | Cloud Agent; no GPU; A6 now executes tests |
+| 2026-09-07 | all local_* eval tools | stub | 20-case corpus; by_tool / by_category rates | pass@end 1.00 (observed); pass@1 0.00 | Cloud Agent; no GPU; 42 attempts; escalated 0.40 |
 
 Columns:
 

docs/roadmap.md

@@ -10,7 +10,7 @@ LAN addresses, or SKUs here.
 | --- | --- | --- |
 | 1 — inference host | Done on the workstation lab | Ollama + starter tags on a private GPU |
 | 2 — MCP bridge | Done | `local-coding-slm` stdio tools; Cursor / Copilot / Claude adapters |
-| 3 — measure | Protocol + expanded corpus + MCP apply gate + CI; live rates still informal | Layered scoring; scripted premium routing/review; no auto-classifier yet |
+| 3 — measure | Protocol + 20-case corpus + stratified stub stats + apply gate + CI; live rates still informal | Layered scoring; scripted premium routing/review; no auto-classifier yet |
 | T12 second NVIDIA host | Blocked on host power | WSL GPU via SSH; see [examples/downstairs-wsl-gpu.md](../examples/downstairs-wsl-gpu.md) |
 
 Finish Phase 3 measurement (layered live rates on the committed corpus)

scripts/run_harness.py

@@ -20,7 +20,8 @@
     run_campaign,
     run_orchestrated_campaign,
 )
-from local_coding_slm.eval.record import summarize, write_jsonl  # noqa: E402
+from local_coding_slm.eval.record import write_jsonl  # noqa: E402
+from local_coding_slm.eval.stats import enrich_summary  # noqa: E402
 
 
 def main() -> None:
@@ -93,7 +94,7 @@ def main() -> None:
         dest.mkdir(parents=True, exist_ok=True)
         write_jsonl(str(dest / "attempts.jsonl"), rows)
         (dest / "summary.json").write_text(
-            json.dumps(summarize(rows), indent=2) + "\n",
+            json.dumps(enrich_summary(rows), indent=2) + "\n",
             encoding="utf-8",
         )
         print(f"wrote {dest / 'attempts.jsonl'}")

src/local_coding_slm/eval/cases.py

@@ -281,6 +281,7 @@ def _run_generated_tests(merged: dict[str, str]) -> None:
         style="Keep the code minimal and preserve type hints.",
         required_top_level=("_normalize_whitespace", "normalize_user"),
         behavior=WHITESPACE_CHECKS,
+        category="extract",
     ),
     EvalCase(
         id="whitespace_extract_vague",
@@ -290,6 +291,7 @@ def _run_generated_tests(merged: dict[str, str]) -> None:
         style="Keep the code minimal and preserve type hints.",
         required_top_level=("_normalize_whitespace", "normalize_user"),
         behavior=WHITESPACE_CHECKS,
+        category="prompt_contract",
     ),
     EvalCase(
         id="multi_file_rename",
@@ -306,6 +308,7 @@ def _run_generated_tests(merged: dict[str, str]) -> None:
             BehaviorCheck("use", "total", ([1, 2, 3],), 6),
             BehaviorCheck("use", "total", ([],), 0),
         ),
+        category="rename",
     ),
     EvalCase(
         id="test_add_execute",
@@ -315,11 +318,27 @@ def _run_generated_tests(merged: dict[str, str]) -> None:
         style="pytest",
         extra_structure=_has_test_functions,
         behavior_fn=_run_generated_tests,
+        category="tests",
     ),
 )
 
 SEED_CASE_IDS: tuple[str, ...] = tuple(case.id for case in CASES)
-CASES = CASES + EXTENDED_CASES
+
+from local_coding_slm.eval.cases_more import (  # noqa: E402
+    EXTRACT_DATACLASS_NESTED,
+    EXPLAIN_MEAN_VAGUE,
+    IMPLEMENT_MEDIAN_FIRST,
+    IMPLEMENT_SLUG_NO_HYPHEN,
+    MORE_CASES,
+    MORE_GOLDEN,
+    REVIEW_DIVZERO_LGTM,
+    RENAME_THREE_PARTIAL,
+    SPLIT_SETTINGS_MONOLITH,
+    TEST_CLAMP_SHAPE_ONLY,
+    TEST_MEDIAN_SHAPE_ONLY,
+)
+
+CASES = CASES + EXTENDED_CASES + MORE_CASES
 CASES_BY_ID = {case.id: case for case in CASES}
 
 GOLDEN_FOR_CASE = {
@@ -328,6 +347,7 @@ def _run_generated_tests(merged: dict[str, str]) -> None:
     "multi_file_rename": MULTI_FILE_GOLDEN,
     "test_add_execute": TEST_ADD_GOLDEN,
     **EXTENDED_GOLDEN,
+    **MORE_GOLDEN,
 }
 
 
@@ -454,4 +474,76 @@ class Fixture:
         False,
         "structure",
     ),
+    Fixture("extract_dataclass_golden", "extract_dataclass", GOLDEN_FOR_CASE["extract_dataclass"], True),
+    Fixture(
+        "extract_dataclass_nested",
+        "extract_dataclass",
+        EXTRACT_DATACLASS_NESTED,
+        False,
+        "structure",
+    ),
+    Fixture("rename_three_golden", "rename_three_files", GOLDEN_FOR_CASE["rename_three_files"], True),
+    Fixture(
+        "rename_three_partial",
+        "rename_three_files",
+        RENAME_THREE_PARTIAL,
+        False,
+        "format",
+    ),
+    Fixture("split_settings_golden", "split_settings", GOLDEN_FOR_CASE["split_settings"], True),
+    Fixture(
+        "split_settings_monolith",
+        "split_settings",
+        SPLIT_SETTINGS_MONOLITH,
+        False,
+        "format",
+    ),
+    Fixture("implement_slug_golden", "implement_slug", GOLDEN_FOR_CASE["implement_slug"], True),
+    Fixture(
+        "implement_slug_no_hyphen",
+        "implement_slug",
+        IMPLEMENT_SLUG_NO_HYPHEN,
+        False,
+        "behavior",
+    ),
+    Fixture("implement_median_golden", "implement_median", GOLDEN_FOR_CASE["implement_median"], True),
+    Fixture(
+        "implement_median_first",
+        "implement_median",
+        IMPLEMENT_MEDIAN_FIRST,
+        False,
+        "behavior",
+    ),
+    Fixture("test_clamp_golden", "test_clamp_execute", GOLDEN_FOR_CASE["test_clamp_execute"], True),
+    Fixture(
+        "test_clamp_wrong_assert",
+        "test_clamp_execute",
+        TEST_CLAMP_SHAPE_ONLY,
+        False,
+        "behavior",
+    ),
+    Fixture("test_median_golden", "test_median_execute", GOLDEN_FOR_CASE["test_median_execute"], True),
+    Fixture(
+        "test_median_wrong_assert",
+        "test_median_execute",
+        TEST_MEDIAN_SHAPE_ONLY,
+        False,
+        "behavior",
+    ),
+    Fixture("explain_mean_golden", "explain_mean", GOLDEN_FOR_CASE["explain_mean"], True),
+    Fixture(
+        "explain_mean_vague",
+        "explain_mean",
+        EXPLAIN_MEAN_VAGUE,
+        False,
+        "structure",
+    ),
+    Fixture("review_divzero_golden", "review_divzero", GOLDEN_FOR_CASE["review_divzero"], True),
+    Fixture(
+        "review_divzero_lgtm",
+        "review_divzero",
+        REVIEW_DIVZERO_LGTM,
+        False,
+        "structure",
+    ),
 )

src/local_coding_slm/eval/cases_extended.py

@@ -460,6 +460,7 @@ def _review_flags_auth(merged: dict[str, str]) -> None:
             BehaviorCheck("report", "summarize", ([], 0, 5), 0.0),
         ),
         max_tokens=1200,
+        category="extract",
     ),
     EvalCase(
         id="extract_shared_parser",
@@ -477,6 +478,7 @@ def _review_flags_auth(merged: dict[str, str]) -> None:
             BehaviorCheck("csv_orders", "order_qty", (" widget , 3 ",), 3),
         ),
         max_tokens=1200,
+        category="extract",
     ),
     EvalCase(
         id="split_pipeline",
@@ -491,6 +493,7 @@ def _review_flags_auth(merged: dict[str, str]) -> None:
             BehaviorCheck("pipeline", "run", ("",), "0"),
         ),
         max_tokens=1200,
+        category="split",
     ),
     EvalCase(
         id="implement_clamp",
@@ -504,6 +507,7 @@ def _review_flags_auth(merged: dict[str, str]) -> None:
             BehaviorCheck("clamp", "clamp", (-2, 0, 5), 0),
             BehaviorCheck("clamp", "clamp", (9, 0, 5), 5),
         ),
+        category="implement",
     ),
     EvalCase(
         id="explain_clamp",
@@ -513,6 +517,7 @@ def _review_flags_auth(merged: dict[str, str]) -> None:
         expect_fences=False,
         required_phrases=("clamp", "lo", "hi"),
         behavior_fn=_explain_mentions_clip,
+        category="explain",
     ),
     EvalCase(
         id="review_login",
@@ -522,6 +527,7 @@ def _review_flags_auth(merged: dict[str, str]) -> None:
         expect_fences=False,
         required_phrases=("none",),
         behavior_fn=_review_flags_auth,
+        category="review",
     ),
 )