CI: reduce test wall time in GitHub Actions and Buildkite

[fluidnumericsJoe]

CI wall time has grown to the point where it shapes what changes are worth making. On #174 the
gfortran-12 coverage job was cancelled twice at its 90-minute timeout, and the second
cancellation was triggered by a commit that changed nothing but documentation. That is not a
one-off: it cost a round of work trimming tests that turned out not to be the cause, and a second
round putting them back after the trim lost real coverage.

This issue is about the runtime itself, in both GitHub Actions and Buildkite. All numbers below are
measured, and I have flagged which are CI-observed and which are local.

The shape of the problem

Observed gfortran-12 coverage durations on one branch, against timeout-minutes: 90:
43, 82, 85, 85, 90 (cancelled), 90 (cancelled), 85 min. The 43-minute run carried more tests than
the 82-minute one. So runner speed dominates, not test count — the job sits close enough to the
cap that a slow draw cancels it regardless of content. Debug jobs land at 54–84 min; release jobs at
16–21 min; Buildkite at ~61 min.

Where the time actually is (289 tests, CTestCostData.txt, gfortran debug build on a
workstation — debug is the relevant configuration since it is the slow one in CI):

	tests	cost	share
total	289	7723 s (129 min)
slowest single test	1	813 s	10.5%
top 10	10	4065 s	53%
≥ 20 s	55	7342 s	95%
< 5 s	217	95 s	1.2%

By label: example 2787 s across 17 tests, serial 4594 s across 226, parallel 1161 s across 46.

The ten worst:

813 s  linear_euler3d_spherical_soundwave_radiation
478 s  linear_shallow_water_2d_constant
421 s  linear_shallow_water_2d_nonormalflow
412 s  linear_shallow_water_2d_radiation
368 s  advection_diffusion_3d_rk4
355 s  ec_advection_3d_rk3
343 s  linear_euler2d_planewave_reflection
332 s  linear_euler2d_planewave_propagation
276 s  advection_diffusion_3d_nostress
268 s  advection_diffusion_3d_rk3

The suite is executed 11 times per pull request. Eight GitHub Actions jobs — linux-gnu-cmake
(gfortran 9/10/11 debug + gfortran-12 coverage) and linux-gnu-multithreaded-cmake (gfortran
9/10/11/12 release) — plus three Buildkite pipelines (MI210, V100, x86 CPU). Every one of them runs
the whole thing.

Optimizations, roughly in order of leverage per unit of risk

1. Run ctest in parallel. Nothing currently does. Every invocation in
.github/workflows/* and .buildkite/* is bare ctest with no -j, so 289 tests run
one at a time on a multi-core runner while make -j $(nproc) is used two steps earlier. This is the
single largest free win.

Prerequisite, and the reason it cannot just be switched on: add_mpi_fortran_tests in both
test/CMakeLists.txt and examples/CMakeLists.txt sets LABELS only — no PROCESSORS property.
ctest -j would therefore schedule many 2- and 4-rank MPI tests concurrently and oversubscribe the
runner, which at best inflates the times it is meant to reduce and at worst makes MPI tests flaky.
PROCESSORS is currently set on 7 tests out of ~48 MPI ones. Setting it in the two helpers is a
small, contained change and should land first.

2. Stop running the full example set in every configuration. example-labelled tests are 2787 s
of the 7723 s — 36% — across 17 tests, and they are integration runs whose value is largely
compiler-independent. Running them in one configuration (say the coverage job, which feeds codecov)
and excluding them elsewhere with -LE example would take a large bite out of seven of the eight
Actions jobs.

3. Trim the heaviest individual tests. One test is 10.5% of the suite and ten are 53%. Most are
long time-integration runs, and it is worth asking per test whether the step count still buys
anything the assertion needs — the tail is already cheap, so this is where the remaining time is.
There is precedent: d88910d4 trimmed the 3-D AMR soundwave family for exactly this reason.

4. Reconsider whether three debug matrix entries need the whole suite. gfortran 9/10/11 debug
exist to catch compiler-compatibility problems, and those surface at compile time or in a smoke
subset. One full debug run plus a build-and-subset for the others would cut a lot. (Related, and
worth deciding separately: linux-amdflang-cmake and linux-nvidia-hpc-cmake are
disabled_manually, so of the four compilers CLAUDE.md requires, CI currently verifies one. Whatever
is done here should leave room to re-enable those rather than assuming gfortran is the whole matrix.)

5. Drop --verbose from the Actions ctest calls. --output-on-failure is already passed and is
what you want; --verbose additionally dumps every passing test's output, which for 289 tests is a
large log for no diagnostic gain.

6. Raise timeout-minutes as a stopgap only. All four build/test workflows use 90. Raising it
would stop the spurious cancellations immediately, but it treats the symptom — worth doing alongside
the real fixes rather than instead of them.

Buildkite specifically

The three GPU/CPU pipelines each build from scratch and run the full suite (~61 min end to end).
Items 1–3 apply unchanged. Two additional questions: whether the MI210 and V100 pipelines need the
entire suite rather than the subset that actually touches device code paths, and whether the x86 CPU pipeline is duplicating what the eight Actions jobs already cover.

Suggested sequencing

PROCESSORS on the MPI helpers → ctest -j everywhere → -LE example on the jobs that do not feed
codecov → then measure again before touching individual test costs. Steps 1 and 2 are mechanical and
should be measured together, because the concentration figures above mean parallelism and example
exclusion interact: with -j, excluding a 46-minute serial block matters less than it looks.

One measurement caveat

The cost table is from a debug build on a 12-core workstation, not a CI runner. Debug is the right
configuration to reason about (release runs the same suite in 16–21 min, so the heavy
time-integration tests are roughly 4–5× slower under -O0 plus bounds checking), but absolute
numbers will differ on a 2- or 4-core runner and the relative concentration is the durable part.
Before and after any change here, Testing/Temporary/CTestCostData.txt from the CI run itself is
the number to compare.