Skip to content

Refresh CI scheduling estimates after runtime improvements - #37

Draft
284km wants to merge 2 commits into
mainfrom
perf/refresh-ci-timings
Draft

284km wants to merge 2 commits into
mainfrom
perf/refresh-ci-timings

Conversation

@284km

@284km 284km commented Nov 7, 2026 •

Copy link
Copy Markdown
Owner

Runtime fixes changed the costs of the 758 CRuby test files, but CI still schedules them using the earlier snapshot. In the completed main run, the eight shards ranged from 708 to 1,384 seconds of supervised execution.

Refresh unittest/timings.json from run 42361097257, preserving its source/compiler/Ruby provenance, and document the measured distribution. Replaying those same observations produces 960 seconds per shard and reduces the four-worker execution estimate from 2,298 to 1,920 seconds (16.4%). This is a scheduling replay, not a measured end-to-end speedup. File coverage, limits and concurrency are unchanged.

Validation: all 758 files are assigned exactly once; the existing CI harness suite passed 45 tests (5 environment-dependent skips). The source report collected all eight shards without infrastructure errors. Its 22 memory-limited files and other compatibility failures remain explicitly reported.

@284km 284km left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 8dfc245 against e45622a. No actionable findings.

The refresh preserves the complete timing inventory and uses the originating run's source/compiler/Ruby provenance. The estimates affect placement only; the workflow, limits, concurrency, and verdict handling are unchanged.

Validation:

  • Downloaded unittest-report and unittest-plan from completed successful run 42361097257. The new snapshot exactly matches its generated timings artifact except for the added run_url; all provenance fields agree with the report and source plan.
  • All 758 keys are unique and unchanged from the base, with finite positive durations. They exactly cover the source manifest and the inventory rediscovered from the pinned Ruby commit.
  • Replanned with the production unittest/ci.py scheduler: all 758 files occur exactly once across eight shards, all weights match, and the recorded timing hash is correct. The old weights also reproduce the original shard assignments.
  • Independently replayed the observations: old shard totals are 707.950–1,383.483 s; refreshed totals are 959.771–960.594 s. Four-worker replay is 2,298.315 → 1,920.384 s, a 16.444% reduction, consistent with the README. This is a scheduling replay, not a new execution-time measurement.
  • The source report confirms 8/8 shards, zero missing files/infrastructure errors, zero WALL_LIMIT files, 22 MEMORY_LIMIT files, and 166 incomplete or zero-test files, matching the documented qualifications.
  • python3 unittest/ci_check.py -v: 45 tests, OK, with five optional real-Ruby fixture tests skipped because their environment variables were unset.
  • git diff --check passed. This PR's harness and planning checks have passed; runtime/reference/Linux jobs remain pending at review time.

No interpreter rebuild or full CRuby rerun was needed for this data-and-documentation change.

Comment thread unittest/timings.json
"schema": 1,
"source_ref": "247032b667ab37c2613e1c77dcb3802dcf3d8681",
"mere_ref": "a697945bb4b88bb4367d6d457e78f6ab54e04ff1",
"source_ref": "278a46cbd707272222759b8161cbb41fd1fa1d7b",

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Positive: Keeping the observation's original source_ref and compiler/Ruby refs preserves provenance across this refresh. I compared all 758 values and these refs with the downloaded report: the snapshot is identical except for the added run URL, and the file inventory is unchanged.

Comment thread unittest/README.md
shards. The observed execution totals ranged from 708.0 to 1,383.5 seconds per
shard with the previous weights. Replaying the same observations with refreshed
weights gives 959.8–960.6 seconds per shard. A four-worker replay falls from
2,298.3 to 1,920.4 seconds (16.4%); this excludes setup, queueing and future

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Positive: The replay qualification is accurate. Using the production partitioner and four-worker scheduling model reproduces 2,298.315 → 1,920.384 seconds (16.444%). The adjacent resource-limit and incomplete-file counts also match the source report, so balanced scheduling is not presented as complete Ruby conformance or a measured end-to-end speedup.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant