Add natural-language benchmark creation and simplify execution

[sumukshashidhar]

Problem and result

Building a benchmark previously required assembling a brittle YAML pipeline, and model/storage failures could silently lose questions or overwrite data. This redesign adds a natural-language frontend and a small Python API while simplifying the existing execution path and making failures explicit.

yourbench create "Test understanding of policy exceptions" \
  --source ./documents --output ./benchmark --model MODEL_ID \
  --base-url http://localhost:8000/v1 --max-tokens 4000 --concurrency 2
yourbench inspect ./benchmark --json
yourbench run ./benchmark

Changes

• Frontend and library: inspectable plan.json/config.yaml; create, run, load_result, and BenchmarkResult; local result inspection without model credentials; per-response token and per-model concurrency controls.
• Runtime: one stage catalogue, dependency/storage preflight, ordered bounded inference, transient retries, sibling cancellation, and owned client lifetimes. No fabricated empty model results. Fresh Hub runs avoid optional reads before repository creation, with five preflight/publication regressions.
• Contracts and storage: one strict JSON decoder and Pydantic validation path; literal custom fields; deterministic correct-option identity; canonical document/chunk provenance; lossless export and rewriting; staged dataset writes with rollback and explicit corruption errors.
• Documentation: consolidated migration guide, Python guide, refreshed CLI/configuration/provider/schema/column/FAQ docs, and six local examples with included fictional Markdown/PDF inputs and offline viability tests.

Migration

See the migration guide for the complete changes and limits. Significant changes include recipe-relative paths, explicit prompt-file errors, removed ingestion upload_to_hub, JSON summary/rewrite responses, removed inferred field/difficulty aliases, choice-index gold for open-ended exports, and JSONL inference metrics replacing duplicate CSV reporting. Generated recipes are local-only; existing YAML can still explicitly publish to the Hub.

This PR does not publish a package release.

Verification

• 250 tests pass on Python 3.12, including actual CLI subprocesses against a local HTTP server, public API execution, offline inspection, interrupted-write/corruption behavior, source/schema preservation, and executable example preflight.
• All five deliberate runtime mutations detected; reproducible procedure in docs/superpowers/plans/2026-09-07-flat-core-verification.md.
• Ruff/format checks, including the repository's CI-pinned Ruff 0.8.4; wheel/source build and public imports from the built wheel.
• Real-model trial: 11 successful calls without retries; 18 questions initially and 21 on rerun; dataset/JSONL parity, source/answer indexing, and all 98 citations verified. Only fictional inputs were sent; credentials and local endpoint details are excluded from this PR.

Limits

The real trial exposed two questions whose single-document evidence omitted rules in the other document. The follow-up adds stage-specific evidence instructions and request regressions; semantic improvement has not been revalidated with a real model. Completion and structural validation do not establish question quality, exact counts, or cost. Storage rollback covers individual writes, not whole-run transactions or concurrent writers. A failed run may retain previous artifacts; configuration-loading failures can leave an older manifest status unchanged.

[sumukshashidhar]

Superseded by #202, which contains the same redesign in a consolidated commit plus the synthetic-URL test and PDF diff fixes. This keeps the secret scanner enabled without retaining the deliberately fake credential-shaped fixture in the PR commit history.