Affected Repositories

• PyAutoFit (primary)

Branch Survey

Suggested branch: feature/hpc-campaign-p2-campaign-core (two stacked PRs)

Implementation Steps

PR 2a — spec, schedulers, lint

1. autofit/campaign/spec.py: Campaign/Stage from campaign.yaml (report §3.2): stages, manifest (0-based line = array index), chunks ≤ max_array_size, resources (cpus, mem_per_cpu, time, partition, gres), after: {stage: aftercorr}, success: {zip_complete, members}, propose-only retry, site: resolved through autonerves.site_config.Site. Validation errors name the field.
2. autofit/campaign/scheduler/base.py: Scheduler protocol (capacity, query, submit, update_pending, cancel) + TaskRecord; fake.py (scriptable outcomes for tests), local.py (subprocess pool), slurm.py (batched sacct --parsable2 --units=M, squeue -r -O, sinfo -N -O; sbatch generation with --dependency=aftercorr, --kill-on-invalid-dep=yes, --mem-per-cpu, --signal=B:USR1@600, PYTHONUNBUFFERED=1). Command runner is injectable, so tests never call Slurm.
3. autofit/campaign/lint.py: partition lint over the stage request + live capacity(): gpu in the partition list with no gres/gpu → refuse unless ≤8 CPUs/task AND throttle ≤ %2 AND no pending gres/gpu jobs; gpu_only partitions never packed with CPU tasks. Every branch tested, including ral,gpu and gpu,ral.
4. CLI validate, submit (lint always on; --dry-run prints sbatch).

PR 2b — collector, digest, retro
5. autofit/campaign/collector.py: scan .status.json records, join with query(), assign collector states timeout/oom/node_fail from sacct; per-stage counts, p50/p95/max wall and RSS against the request, CPU efficiency, idle capacity, ETA, disk used/forecast, drift flags.
6. autofit/campaign/digest.py: digest.json + digest.md ≤2 KB (size-capped like status/writer.compact); failed_<class>.txt; resubmit proposals sized from the failed tasks' own measurements (timeout → p95 + 25 %, OOM → peak × 1.5), capped by site limits, never auto-submitted.
7. autofit/campaign/retro.py: sacct --parsable2 parser (array ids, .batch/.extern steps folded into the parent job, M-suffixed memory, D-HH:MM:SS times, --duplicates), WorkDir/JobName filters → the same digest.
8. CLI status, resubmit, retro --sacct <dump> [--status-root] [--workdir-prefix].

Tests

• FakeScheduler end to end with injected timeout, oom, exception and missing-seed tasks → right classes, lists and proposals
• digest ≤2 KB
• lint matrix
• synthetic sacct retro fixture
• sbatch text golden tests
• no subprocess call at import

Witness W2

retro over Workspace/Science/active/euclid_dr1/inspect/sacct_retrospective_2026-10-28/*.psv (134k rows) filtered to the DR1 WorkDirs; digest + figures (dataviz skill) in Workspace/Science/scratch/hpc_campaign_p2_w2/. Never committed.

Key Files

• autofit/campaign/ (new): spec.py, scheduler/, lint.py, collector.py, digest.py, retro.py, __main__.py
• reads autofit/non_linear/status/states.py, autonerves/site_config.py

Epic: hpc-campaign
Phase: 2
Blocked-by: phase 1 (COMPLETE 2026-10-29)
Concurrent-with: results-library phase 1 (ruling F11 — two PyAutoFit worktrees; this task owns autofit/campaign/ only and imports non_linear/status/states.py read-only)
Consequence: judge
Autonomy: human-required
Witness: W2 — python -m autofit.campaign retro --sacct over the F4a dump Workspace/Science/active/euclid_dr1/inspect/sacct_retrospective_2026-10-28/sacct_jnightin_20260901_20261028.psv filtered to the DR1 WorkDirs; digest + figures (dataviz skill) in Workspace/Science/scratch/hpc_campaign_p2_w2/; never committed to a public repo.

HPC campaign phase 2 — campaign core, scheduler adapters, collector, partition lint, retrospective mode

Original request (human, 2026-10-30): "Ctoninue the PyAutoLEns II HPC runs and Results"

Ledger: draft/research/autofit/hpc_campaign_epic.md (Hard constraint; rulings F4, F5).
Report: hpc_campaign_epic_report.md §3.2 (campaign.yaml, digest schema).
Lane prompt: Workspace/Science/scratch/pyautolens_ii_campaign_results_scoping_2026-10-28/PROMPT_hpc_campaign.md §Phase 2.

@PyAutoFit scope (autofit/campaign/, stdlib + PyYAML, no Slurm at import):

• campaign.yaml loader/validator (stages, manifest, chunking ≤ max_array_size, resources, after,
  success, propose-only retry (F5), site: from PyAutoNerves site.yaml).
• Scheduler protocol + FakeScheduler, LocalScheduler, SlurmScheduler (batched sacct/squeue/sinfo,
  sbatch generation with aftercorr, --kill-on-invalid-dep=yes, --mem-per-cpu, --signal=B:USR1@600).
• Partition lint: refuse gpu in a partition list for a no-gres script unless all three exemption
  conditions hold at submit time; gpu_only partitions never packed with CPU tasks.
• Collector → digest.json + digest.md (≤2 KB), failed_<class>.txt, propose-only resubmits.
• Retrospective mode campaign retro --sacct <dump> [--status-root].
• CLI python -m autofit.campaign {validate,submit,status,resubmit,retro}.

Done: FakeScheduler end-to-end with injected timeout/oom/exception/missing-seed; digest size test; lint
tests for every branch incl. ral,gpu and gpu,ral; synthetic sacct retro; W2 witness (F4a authorised).