Repository navigation
ci: categorise the tests into tiers and run each tier where it belongs #1958
Description
An audit of every test module at b2cd14078. Timings are from Unit Tests run 39331834560. It proposes a target tier for each kind of test. Builds on #1954 / #1956.
Proposed tiers
| Tier | Runs on | What belongs |
|---|---|---|
ALL |
every CPython leg, KTLO, 3.14t, GraalPy | unit, compat (version-gated behaviour), simple integration on generated captures |
LATEST |
latest CPython only | complex integration (multi-engine parity, large captures, subprocess), repo-text / workflow / docs pins, round-trip sweeps |
PERIODIC |
cron only | drift checks against external state; none found among the tests (the cron-* workflows already do this) |
PR |
pull requests only | diff-dependent checks; none found among the tests (Changelog drift is already its own job) |
Of the module classifications, 385 land in ALL, about 30 of them borderline, and 144 in LATEST, about 40 borderline. The LATEST ones carry about 710 s of 3.15 test time. Today most of them also run on every leg.
Where today's placement disagrees
- The
*roundtrip*glob (ci(python): the GraalPy experimental leg reaches 26% of the suite before its 25-minute cap #1732) is a name rule. It sends 30 small hand-picked round-trip unit modules to 3.15 only, e.g.test_tcp_ns_flag_roundtrip_unitandtest_from_dict_roundtrip_unit. It also misses real sweeps under other names, e.g.test_dispatch_reachability_unitat 24–55 s andtest_misc_short_read_1470_runtime. - ci(unit-tests): run platform-independent project pins once per push (#1954) #1956's pin rule is conservative. It keeps on every leg 36 project modules whose answer does not vary by interpreter: those that load
conf.py, testutil/scripts, or check workflow shapes. That is about 99 s per leg. Pins outsidetests/projectare not covered at all, e.g.tests/corekit/test_sentinel_exports_unit.pyandtests/test_docstring_contract.py. - Tests that never run on push or PR:
tests/foundation/engines/test_runtime_engines.pyneeds dpkt, scapy and pyshark together, and no job installs all three, so it runs nowhere.- Scapy-gated tests outside the
Engines (Scapy)path list run only in the Saturday/releasegate. - 11 vendor regeneration modules need
html5lib, which is a documented choice (tests: seven more skipUnless gates have never executed on any CI path (106 methods) #738, ci: rebuild engine-tests as a Python × engine matrix with three cell outcomes #849). - Sphinx-gated docs tests run in no pytest job.
- Misfiled tiers.
tests/integration/test_module_loading.pyand two*_runtime.pymodules read no capture. The parity job runs the round-trip sweeps on 3.10 and 3.11, against ci(python): the GraalPy experimental leg reaches 26% of the suite before its 25-minute cap #1732. - Cost. A push costs about 300 runner-minutes. GraalPy runs into its 110-minute cap on most runs, about a third of that. PyPy runs no tests.
Decisions
- A. Adopt the four tiers. Should one executable rule assign them, extending ci(unit-tests): run platform-independent project pins once per push (#1954) #1956's
tests/_tiers.pybeyondtests/project, or an explicit per-module marker? - B. Should hand-picked round trips run on
ALL, and sweeps be found by behaviour rather than name? - C. Should the dark tests get a job: one latest-CPython leg with every engine plus Sphinx, on push or on PR?
- D. GraalPy: keep it as is, trim it to the
ALLtier, or move it to cron? - E. Should PyPy get an
ALLleg?
The per-module tables (path, kind, tier, current placement, seconds, reason) are available, and a follow-up comment can carry them. A separate pass is checking the historical logs of the non-blocking legs that end green, for real failures to fix.
Activity
JarryShaw commentedon Oct 20, 2026
A. I'd prefer executable rule assignment for single-place tracking but also comes with per-module marker for explicity.
B. I'd say selected ones should run on
ALLas part of the quality assurance.C. yes, and assign them to the corresponding tiers.
D. depending on how we've tested it. if GraalPy support is generally validated, we can now promote it from experimental classes, free-threaded CPython as well.
E. I'd say ALL implementations should get the same minimal tier of tailored test suite for quality assurance purposes. But latest CPython will have the whole suite for sanity check and gatekeeping.
JarryShaw commentedon Oct 20, 2026
The plan, in four PRs, each stacked on #1956 until it merges:
tests/_tiers.pybecomes the single source for every module undertests/. A module-level__tier__constant ('ALL','LATEST', …), read by AST, overrides the executable rule where the rule's guess is wrong or worth stating outright. A constant works under plainunittesttoo, which a pytest marker would not.*roundtrip*name glob goes. The hand-picked round trips run onALL; the real sweeps are markedLATEST.html5lib, and runs every tier. That brings the dark tests in.ALLtier on every implementation (E). Add PyPy legs that run theALLtier, as the other non-latest legs do.Starting on 1.
ci(unit-tests): assign every test module a tier and run each tier whe…
JarryShaw commentedon Oct 20, 2026
After the review of #1971, the four
*_edge_roundtrip_unittables move toALL, since their hand-written rows serve ruling B's purpose.test_short_read_roundtrip_unitand the deadline harness test stayLATEST. This lands as a test-only follow-up.test(protocols): run the hand-written edge-case round-trip tables on …
JarryShaw commentedon Oct 20, 2026
Blocked until #1973 merges. Phase 2 (the whole suite on latest CPython) rewrites the same
unit-tests.ymltest commands that #1973 wraps with the file-size cap.24 remaining items
JarryShaw commentedon Oct 21, 2026
3.14t findings, from
experimental.ymlonmain(2026-10-19 tod419d8c39). Each run is judged by its tests step's pytest summary, since the job itself always ends insuccess.a1cd9efbc, or 19 counting from ci(experimental): find what drives the 3.14t leg out of memory #1964's fix at58fc347b3. The latest, job 119413626555 on 3.14.9, gave 2173 passed and 80 skipped in 204 s. Before that, 3 runs failed the test(vendor): the free-threaded lxml GIL warning breaks the empty-stderr check on 3.14t #1961 test andc8c34a86flost a worker to the OOM killer.RLIMIT_AS.a1cd9efbc): lxml's GIL warning appearing in a child process's stderr.58fc347b3): the collector freed cycles late. Peak memory went from about 13 GiB to at most 3.7 GiB.lxml.etree, pulled in through bs4, turns the GIL back on in every run. In three sampled runs, one worker did so about 8% of the way through the tier and the other at 97–99%. So roughly half the tier runs with the GIL on. No other module does this, and nothing setsPYTHON_GIL.The record supports promoting 3.14t. That leaves item 5 above: set
PYTHON_GIL=0on the leg, or keep lxml out of it, if the tier should run truly free-threaded.JarryShaw commentedon Oct 21, 2026
lxmlfrom 3.14t tests.JarryShaw commentedon Oct 21, 2026
Plan from these rulings:
experimentaltosupported, joinsunit-tests.ymltest, and gets aCompat Python 3.14tleg. I'll update the ruleset myself if the new leg needs its own required context.PYTHON_GIL=0, plus an assertion thatsys._is_gil_enabled()is false. Iflxmlis unsafe or crashes with the GIL off,lxmlcomes out of the 3.14t install, and the tests that need it skip there.util/python_versions.py. It lands after ci: run the curated ALL tier on PyPy, split into separate pytest runs (#1958, #2005) #2006.load()once about 5 green runs in a row are onmain.ci: run the curated ALL tier on PyPy, split into separate pytest runs (…
ci: run the curated ALL tier on PyPy, split into separate pytest runs (…
ci: promote free-threaded 3.14t to supported, with the GIL off (#1958)
ci: run the curated ALL tier on PyPy, split into separate pytest runs (…
ci: promote free-threaded 3.14t to supported, with the GIL off (#1958)
JarryShaw commentedon Oct 22, 2026
Phase 4 is dropped: 3.14t and GraalPy both stay experimental (ruling on #1967 at 02:05Z), so #2024 is closed and the ruleset is unchanged. Phases 1–3 landed in #1971, #1976, #1978 and #2006. Closing; I'll reopen this if promotion comes back.