Skip to content

ci: PyPy workers on the ALL tier grow until they are OOM-killed #2005

Closed

Description

@JarryShaw

On PyPy, the unit ALL tier's pytest workers grow to roughly 3.5–4.5 GB each.

  • Symptom: in a 14 GB container at -n 4, four workers were OOM-killed, at different tests on each run. Capping the heap at 3 GB only turned the kills into MemoryErrors.

  • Expected on CI: a 16 GB GitHub runner is likely to hit the same limit, so the PyPy legs added for ci: categorise the tests into tiers and run each tier where it belongs #1958 phase 3 need containment before they can block.

  • Found on PyPy 8.0.1, run in an Ubuntu 24.04 container.

  • Possible fixes:

    • split the tier into several pytest processes, so each worker starts fresh;
    • recycle workers after a fixed number of tests;
    • lower -n on PyPy.

    Whichever is chosen, measure peak RSS per worker before and after.

Activity

  1. added
    bugIssues reporting a defect (set by the bug report template; a default, not an assessment)
    fixPull requests that fix a defect (fix: subject prefix)
    pendingAccepted and unstarted - nobody is on it and nothing blocks it
    on Oct 20, 2026
  2. added
    ciPull requests that change CI or workflow configuration (ci: subject prefix)
    wipWork in flight - a covering PR is open or an agent is actively on it
    and removed
    fixPull requests that fix a defect (fix: subject prefix)
    pendingAccepted and unstarted - nobody is on it and nothing blocks it
    on Oct 20, 2026
  3. JarryShaw commented on Oct 20, 2026

    @JarryShaw
    OwnerAuthor

    The fix is folded into draft #2006 (d4db362e0). The growth is steady across tests, not a handful of heavy modules, so lowering -n or setting PYPY_GC_MAX cannot help. A PyPy leg instead runs the tier as separate pytest invocations, each with fresh workers. In a 14 GB container, workers now peak at 2.6 GB and the container at 8.5 GB, with no OOM kills.

  4. removed
    wipWork in flight - a covering PR is open or an agent is actively on it
    on Oct 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugIssues reporting a defect (set by the bug report template; a default, not an assessment)ciPull requests that change CI or workflow configuration (ci: subject prefix)

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions