Repository navigation
rebase: reap attempt-scoped test chains and give module tests a real budget - #431
Merged
Merged
Conversation
…budget
The v0.33 campaign (run-20261024-235522-d35da7) blocked at the test loop
with GpuLockTimeout. Module agents had started pytest chains via setsid;
those outlived their sessions, kept the run's GPU lock for ~7 hours after
the engine finished wave 1, and the lock's liveness check (kill -0) also
treated an unreaped zombie owner as alive.
- rebase_engine/processes.py: cleanup_run_processes() kills same-user
processes carrying BOTH run markers (IMX_RUN_ID + IMX_LOG_DIR), keyed by
process birth time so a reused PID is never a bystander kill. Called
when a module/debug attempt ends (including normal return) and around
the checkout-lock finalizer.
- testing/process_tree._alive(): a Z/X state in /proc/<pid>/stat is dead;
gpu_lock reuses it so a zombie can no longer hold the mutex.
- Module test budget: adapters declare rebase.testing.module_timeout_sec
(vllm-omni: 21600 — the offline inference suite needs ~4h, so the old
7200s wrapper could never reach a terminal result). The value flows to
TEST_TIMEOUT_SEC in the agent shell env, the prompt template
({MODULE_TEST_TIMEOUT_SEC}) and the harness budget note.
- Settings.rebase_harness_timeout_sec replaces the hard-coded 7200s
session budget passed to ModuleRunConfig.
- providers/zcode: write mcp.servers.*.timeoutMs from the session budget;
ZCode's 30s default abandoned bridge tool calls that were still running.
- Prompt template: drop the obsolete 30s-polling / detached-chain guidance
and tell agents the attempt's background processes are reaped on return.
Tests: test_rebase_process_lifecycle.py (real zombie owner, detached
orphan reaped while other runs' processes survive), zcode timeoutMs test,
harness stub gains the new field.
Co-authored-by: Cursor <cursoragent@cursor.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
…le test budgets Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The vLLM-Omni v0.33 alignment campaign (
run-20261024-235522-d35da7) blocked at the test loop withGpuLockTimeout: GPU lock timeout after 3600s. Two engine defects combined:setsid. Those chains outlived their harness sessions, kept the run's GPU lock for ~7 hours after wave 1 had finished, and the lock's liveness check (kill -0) also counted an unreaped zombie owner as alive.model_executormodule could never finish:tests/e2e/offline_inference/needs ~4 h wall-clock, but the per-commandTEST_TIMEOUT_SECwas hard-coded to 7200 s, so the suite was SIGKILLed at ~50 % on every attempt and never produced a terminal result.What
rebase_engine/processes.py(new):cleanup_run_processes()reaps same-user processes that carry both run markers (IMX_RUN_IDandIMX_LOG_DIR), keyed by process birth time so a reused PID can never become a bystander kill. Called when a module or debug attempt ends (including a normal return) and in the checkout-lock finalizer.testing/process_tree._alive(): aZ/Xstate in/proc/<pid>/statis dead;gpu_lockreuses it so a zombie can no longer hold the mutex.rebase.testing.module_timeout_sec(vllm-omni:21600). It flows toTEST_TIMEOUT_SECin the agent shell env, the prompt template ({MODULE_TEST_TIMEOUT_SEC}) and the harness budget note appended to each attempt.Settings.rebase_harness_timeout_secreplaces the hard-coded 7200 s session budget passed toModuleRunConfig.providers/zcode: writesmcp.servers.*.timeoutMsfrom the session budget; ZCode's 30 s default abandoned bridge tool calls that were still running.Tests
test/test_rebase_process_lifecycle.py(new, real processes): an unreaped zombie lock owner is displaced; a detachedsetsidorphan carrying the run markers is reaped while processes of other runs survive.test_provider_zcode.py::test_bridge_timeout_covers_long_running_module_tools.test_rebase_harness_backend.pystub gains the newtest_timeout_secfield.test_adapter_knowledge test_assembly test_provider_zcode test_rebase_harness_backend test_rebase_process_lifecycle test_shell_golden test_testing_substrate→ 207 passed, 1 skipped; the vllm-omni module prompt renders withTEST_TIMEOUT_SEC=21600.Notes
This is the engine fix the restarted v0.33 campaign is running with (target vLLM
0232bb3060, backendcursor-agent/grok-4.7-high). No knowledge files are touched.