[Jammy2211]

Summary

PR 2b of hpc-campaign phase 2 (#1740). Stacked on #1741; merge that first.

• Collector (collector.py): one sacct call joined with the fits' status records, matched by the job and task ids each fit recorded. The scheduler decides timeout/oom/node_fail/cancelled; otherwise the status record gives the state and failure class. It also runs the stage success checks and reports manifest objects that were never submitted.
• Digest (digest.py):
  • digest.json holds the full summary; digest.md is at most 2 KB; tasks.jsonl has one row per task; failures go in failed_<class>.txt.
  • It reports per-stage counts, p50/p95/max wall time and RSS against the request, CPU efficiency, and CPU-hours split into successful vs failed or repeated, plus disk, ETA and drift flags.
• Proposals (proposals.py), propose-only per ruling F5:
  • timeout → p95 + 25 %; OOM → peak × 1.5; both capped by the site, and the retry maximum is respected;
  • Python failures are never resubmitted;
  • a task that already ran to the partition's maximum time is told to resume from its checkpoint instead.
  • A human acts on a proposal with submit --proposal <file>.
• Retrospective mode (retro.py): builds the same digest from a sacct --parsable2 dump alone. Tasks are selected by WorkDir prefix and job-name regex, plus an --other-stage bucket.
• CLI: status, resubmit, retro, submit --proposal.

Witness W2 (ruling F4a, read-only, no compute)

retro over the DR1 sacct dump: 67k rows in 13 s → 61,078 Slurm tasks.

• Outcomes: 56,738 completed, 3,254 timeout, 550 failed (unrecorded, from before status records existed), 524 cancelled, 12 node_fail, 0 OOM.
• CPU-hours: 709.8k allocated, 505.9k of it in tasks that completed (so 29 % went on failed or repeated work).
• Memory: p95 peak use was 2–4× below the request in every stage.
• The 2026-09-30 incident is visible: 568 CPU-only tasks ran on gpu.

Digest and figures are in Workspace/Science/scratch/hpc_campaign_p2_w2/, which is never committed. Counts are per Slurm task, not per lens.

Decisions taken (please check)

1. cancelled/requeued/preempted stay campaign-level strings; states.py is unchanged (results-library owns its edits in this window). A failure with no status record is classed unrecorded. A possible follow-up is RunState.CANCELLED.
2. Only digest.md is capped at 2 KB; the JSON carries the full summary.
3. autofit.campaign does not re-export submit/retro functions (they'd shadow the submodules); import them from the submodules.

Test plan

• test_autofit/campaign (106): an end-to-end FakeScheduler run with injected timeout, OOM, exception and missing-seed tasks giving the right classes, lists and proposals; the digest size cap; a synthetic sacct retro.
• Full PyAutoFit suite (see comment).

Refs #1740

🤖 Generated with Claude Code

[Jammy2211]

Full PyAutoFit suite (PyAuto313, local): 2a 4,082 passed / 2b 4,098 passed (2 skipped, 1 xfailed). Smoke autofit (scripts + notebooks) green on the 2b head. Heart STALE (release validation only) acknowledged by the human for this ship.