[Jammy2211]

Summary

hpc-campaign phase 3 (#1745, epic ledger draft/research/autofit/hpc_campaign_epic.md). It adds planning before submission, in autofit/campaign/ only:

• evidence.py: readers for profiling-summary@3, walltime-validation@1 and inference-summary@2, by path or URL. A stage declares opaque match keys under evidence: in campaign.yaml.
  • Evidence grades on every number: measured-current / provisional / analogue (naming the key that differs) / superseded / estimated / unsupported.
  • A record is measured-current only on exact keys. A profiling record also needs a current, not-superseded Pulse designation for its own keys and setup.
  • "optimal" is never written without such a designation, and none exists today.
• plan.py: python -m autofit.campaign plan writes plan.md + plan.json. It proposes CPUs, memory, wall time, throttle, ETA and disk from a pilot digest and/or the evidence. Every number is graded and carries its record ids, and a test walks plan.json to enforce that.
• pilot.py: campaign pilot writes a stratified 1–2% pilot.json, with the submit --indices command left unrun. Nothing is submitted.

Heart RED override (development only)

Heart was RED at the start of this task (2026-10-30): release validation FAILED (stage integrate); PyAutoHeart/workflows/Release Integrate failure; PyAutoNerves test_fitsable.py::test__output_to_fits red; unit-test timing 1 regression (>3× baseline), 14 slow; autogalaxy red; release validation NOT release_ready (v2026.10.30.1.dev85101); autolens_profiling red; worktree drift 1 dirty. The human authorized the Human override for Heart RED (development only) live for hpc-campaign phases 3 and 4 ("Override, proceed"). This PR does not touch any RED item and does not fix Heart. Merging still needs green CI and a separate /prm; it permits no release.

Decisions taken (please check)

1. CPU right-sizing: below a median CPU efficiency of 0.6, the plan uses the largest power of two not above the CPUs actually used. Wall times measured at a different core count are scaled by measured/requested cores, graded estimated, and flagged cores_mismatch.
2. Memory: the pilot's maximum RSS × 1.25, rounded to 4 GB steps. Wall time: p95 + 25% (timeouts counted at their limit), in 15-minute steps, flagged censored when the p95 reaches the timeout limit, and split by size bin when a size_field is given.
3. When several records match: the largest value among those with the best grade wins, ordered deterministically. All contributing ids and the spread are recorded.
4. Pilot numbers, including the previous DR1 run used as a pilot, are graded estimated (pilot), never measured-current.
5. Throttle is 50% of the stated idle CPUs (given offline with --idle-cpus). ETA is labelled an upper bound, with stages run one after another.

Test plan

• test_autofit/campaign: 128 pass (22 new), including the DR1 repack reproduced from a sacct fixture (CPUs 4/8/8 exact; memory within one 4 GB step: vis_pix 16 and SED 8 GB exact, vis_lp 8 vs 12 GB) and the grade matrix.
• Full PyAutoFit suite: 4,120 passed (local, PyAuto313). Smoke test on the autofit workspace: 10 passed, 0 failed.
• Independent review: FINDINGS. The fix commit 32a9305 addresses:
  • cores mismatch;
  • first-match-wins evidence choice;
  • the cost-per-evaluation fallback;
  • ungraded pilot and size-bin numbers;
  • the designation scope check.
• Witness W3: a graded 5,000-lens Euclid VIS plan in Workspace/Science/scratch/hpc_campaign_p3_w3/ (not committed).
  • vis_lp 4 CPUs / 8 GB / 5:00; vis_pix 8 CPUs / 20 GB / 10:45; SED 2 CPUs / 8 GB / 4:00 (censored, so the pilot must run at a longer limit).
  • ≈108k CPU-h, ETA ≤ 63.5 h.
  • Grades: 0 measured-current, because there are no Euclid walltime or inference records yet. The evidence gaps are listed in its README for the likelihood-lifecycle ledger.

Refs #1745

🤖 Generated with Claude Code