Skip to content

ci: run the GraalPy leg's tier in shards so no worker outlives its heap (#1963) - #2012

Merged
JarryShaw merged 1 commit into
mainfrom
ci/1963-graalpy-split-runs
Oct 21, 2026
Merged

JarryShaw merged 1 commit into
mainfrom
ci/1963-graalpy-split-runs

Conversation

@JarryShaw

Copy link
Copy Markdown
Owner

What is the purpose of your pull request?

  • ci — workflows or build tooling

Description of your pull request and other information

On main's GraalPy leg (job 118912680486), nine workers went down. Each one's Java heap grew across tests until it reached the --vm.Xmx2g cap (#1753). This PR runs the leg's tier as a series of pytest invocations, each with fresh workers.

The shard boundaries are chosen so that no shard's heap gets near the cap. A shard is a directory or a glob of one directory's entries, and every later run ignores what earlier runs took. A whole-directory split was not enough: tests/corekit alone reached 2028 MB.

Measured locally on GraalPy 25.4.4 (-n 3, Xmx2g):

Runtime Workers down Peak live heap
Whole tier 3559 s 8 2038 MB
Sharded 1879 s 0 1303 MB

How it runs. util/pytest_shards.py runs inside the step's existing timeout, prlimit, scope and watcher, so those caps still cover the whole tier. On SIGINT it lets the current run write its JUnit file and starts no further runs. Each shard writes its own junit-<shard>.xml.

Unchanged:

  • the heap flags, -n, the summaries and continue-on-error;
  • the 3.14t leg.

Tests. tests/project/test_graalpy_shards_1963.py checks, through real pytest collection, that the shards cover every test of the tier exactly once. It fails against main (6 failed).

Part of #1963.

…ts heap (#1963)

- util/pytest_shards.py: runs a pytest command once per shard, a
  directory or a glob of one directory's entries, then once over the
  rest; each run ignores what earlier shards took and starts fresh
  xdist workers, writes its own JUnit file, and no later run starts
  after SIGINT. Standard library only, on /usr/bin/python3.
- experimental.yml `experimental` / `Run unit tests`: GraalPy's leg
  sets SHARDS and hands its pytest command to the script inside the
  one timeout, file-size cap, scope and watcher; 3.14t runs pytest
  once, as before. `Upload test timings` takes junit*.xml.
- Pins follow the step; tests/project/test_graalpy_shards_1963.py runs
  the step and the script, and checks the shards partition the tier.
- workflows.rst: the shard bullet and the Workflow Scripts row.

Local GraalPy 25.4.4 at -n 3: whole tier 3559 s, 8 workers down, heap
at the 2 GiB cap; sharded 1879 s, none down, largest heap 1303 MB.
@JarryShaw JarryShaw added ci Pull requests that change CI or workflow configuration (ci: subject prefix) review: running A cross-review is in flight against the current head - no verdict yet labels Oct 21, 2026
@JarryShaw JarryShaw moved this to In review in PyPCAPKit Oct 21, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

GOOD TO GO at 7a910c9aa. A Sonnet review confirmed:

  • Coverage. The pin collects the tier unsharded and through util/pytest_shards.py, then compares the sorted node ids, so a lost or doubled test fails it. Nested directories fall through to the directory shard or the remainder, and tests/conftest.py is unaffected.
  • Exit status. timeout's 124/137 and the PIPESTATUS handling are unchanged, and pytest.log still receives every shard, so the crash and MemoryError counters see all of them. 3.14t runs pytest once, as before.
  • Tests. The 1963, 1740, 1938 and 1732 pins pass.

One residual risk: a SIGINT landing between two shards is honoured only after the next shard starts, so that shard would run until the SIGKILL. The window is tiny.

@JarryShaw JarryShaw added review: good-to-go Cross-review at the current head says ready; CI state is separate and removed review: running A cross-review is in flight against the current head - no verdict yet labels Oct 21, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Coverage: 91.55% (unit tier, Python 3.15, 7a910c9aa, Unit Tests run success)

Package Statements Missed Branches Partial Cover
pcapkit (top level) 104 4 20 2 95.16%
pcapkit/const 19597 881 2618 867 92.11%
pcapkit/corekit 2542 71 810 37 96.54%
pcapkit/dumpkit 276 3 100 3 98.40%
pcapkit/foundation 2930 23 1022 25 98.79%
pcapkit/interface 138 4 52 3 96.32%
pcapkit/protocols 17501 222 4612 187 98.13%
pcapkit/toolkit 872 36 298 10 95.21%
pcapkit/utilities 523 2 158 4 99.12%
pcapkit/vendor 4431 1995 1018 167 51.73%

Per-file detail: the coverage-html artifact of this run.

@JarryShaw
JarryShaw merged commit c306e6c into main Oct 21, 2026
60 checks passed
@github-project-automation github-project-automation Bot moved this from In review to Done in PyPCAPKit Oct 21, 2026
@JarryShaw
JarryShaw deleted the ci/1963-graalpy-split-runs branch October 21, 2026 14:06
@JarryShaw JarryShaw removed the review: good-to-go Cross-review at the current head says ready; CI state is separate label Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci Pull requests that change CI or workflow configuration (ci: subject prefix)

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant