Skip to content

ci(util): use requests for the scripts' HTTP fetches and psutil for vendor_test_gate's memory polling #1894

Closed

Description

@JarryShaw

On #1885 the owner asked for the repo's scripts to use requests and bs4[html5lib], and pexpect or psutil where they read a subprocess's output. #1885 moves util/conda_vendor.py --refresh-pins to requests; this covers the other scripts.

requests, replacing urllib.request:

  • util/ci_timing.py: urlopen at :437-446, Link-header pagination at :412. ci-timing.yml runs it on a bare interpreter and installs nothing (:57), so that job needs setup-python and an install step.
  • util/update_python_versions.py: fetch() at :179-191. The detect job in cron-python-versions.yml installs only PyYAML (:61-62).
  • examples/generators/pcapng.py: _download and its Retry-After backoff at :832-917. Every leg that runs it already installs plain requests through the test extra, but unit-tests.yml:473-476 says it needs only the standard library, so update that comment. Keep the offline fallback and the retry behaviour.

markdown-it-py, replacing the pipe-splitting in util/update_python_versions.py parse_images() (:245-268), which reads the runner-images README's image table. An escaped pipe or a pipe inside a code span would shift its columns; read the table through a GFM-table parser instead (added after the owner's follow-up on #1885). It runs in the same detect job as fetch().

lxml, replacing xml.etree.ElementTree in util/ci_timing.py parse_junit() and util/pyshark_encap_map.py's PDML read, for its strict parsing (the owner's preference on #1885). Use a parser that resolves no entities and has no network access. lxml has wheels for CPython 3.8–3.15 (free-threaded included) and PyPy 3.9–3.11, none for GraalPy: add it to the test extra with a marker excluding GraalPy, gate its tests there, and install it in ci-timing.yml.

psutil, replacing the ps -A -o pgid= -o rss= polling in util/vendor_test_gate.py:157-171. psutil is in no extra, and tests/project runs this script's tests on every leg, including KTLO and experimental. So either confirm that psutil installs on every one of those legs, or keep ps as the fallback.

The tests that stub urllib move with their scripts: tests/project/test_ci_timing.py and tests/project/test_pcapng_download_retry_1658_unit.py.

Out of scope:

  • util/project_status.py stays on gh. Its job runs on pull_request_target, holds PROJECT_TOKEN, and installs nothing by design.
  • bs4 and pexpect have nothing to replace: no script outside pcapkit/vendor parses HTML, and none drives an interactive program.

Activity

  1. added
    ciPull requests that change CI or workflow configuration (ci: subject prefix)
    pendingAccepted and unstarted - nobody is on it and nothing blocks it
    wipWork in flight - a covering PR is open or an agent is actively on it
    and removed
    pendingAccepted and unstarted - nobody is on it and nothing blocks it
    on Oct 17, 2026
  2. JarryShaw commented on Oct 17, 2026

    @JarryShaw
    OwnerAuthor

    I suppose these should also go into the test extras and/or Pipfile dev deps?

  3. JarryShaw commented on Oct 17, 2026

    @JarryShaw
    OwnerAuthor

    They are in the test extra already. #1896 adds all three there:

    • psutil, on CPython only;
    • markdown-it-py;
    • lxml, everywhere except GraalPy.

    The Pipfile's [dev-packages] and the dev extra don't list any of them yet, so #1896 will add all three to both, with the same markers. requests is already in both.

  4. JarryShaw commented on Oct 17, 2026

    @JarryShaw
    OwnerAuthor

    Correction to my reply above: #1896 adds them to the Pipfile's [dev-packages] and the test extra, but not to the dev extra.

    process.rst defines dev as what pylint, mypy and autodoc must resolve for pcapkit/. That is why requests and bs4 are there: pcapkit/vendor imports them. Only the scripts under util/ and examples/ import these three, and the toolchain does not analyse those.

  5. removed
    wipWork in flight - a covering PR is open or an agent is actively on it
    on Oct 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    ciPull requests that change CI or workflow configuration (ci: subject prefix)

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions