Graceful SIGINT/SIGTERM shutdown, resilient tool bridge, wrapper signal receipts

adapters/vllm_omni/rebase/templates/module_rebase.live.prompt.tmpl

@@ -239,13 +239,35 @@ every later call. Therefore:
 - Use `imx-omni-pytest …` with `TEST_TIMEOUT_SEC={MODULE_TEST_TIMEOUT_SEC}` for
   every required verification command. Keep the selected scope, assertions and
   granted GPU geometry; do not shrink or extend the test budget.
-- Never block a tool call on a long wait (`sleep 480 && tail …`). For anything
-  longer than a short command, launch detached with stdin/stdout/stderr
-  redirected and a PID file, e.g.
-  `setsid -f sh -c 'echo $$ > "$1"; shift; exec "$@"' sh "$pidfile" imx-omni-pytest ... </dev/null >"$log" 2>&1`
+- Never block a tool call on a long wait (`sleep 480 && tail …`), and never run
+  a verification command in the foreground of a tool call: a foreground run is
+  killed at the tool's deadline together with its process group. Launch every
+  `imx-omni-pytest` DETACHED — own session, stdin from /dev/null, stdout/stderr
+  to a log, PID saved — in ONE tool call that returns immediately:
+  ```
+  log=$IMX_LOG_DIR/tests/detached-<name>.out; pidfile=$IMX_LOG_DIR/tests/detached-<name>.pid
+  setsid -f sh -c 'echo $$ > "$1"; shift; exec "$@"' sh "$pidfile" imx-omni-pytest ... </dev/null >"$log" 2>&1
+  sleep 1; cat "$pidfile"; ps -o pid,sid,pgid,stat,etimes,cmd -p "$(cat "$pidfile")"
+  ```
   (a plain `nohup ... &` is insufficient: the non-interactive bridge shell waits
-  for that job). Then poll the saved wrapper PID with short calls, at most one
-  check per tool call and under 20 seconds each including any sleep.
+  for that job). The `ps` line is the detach proof: the wrapper's `sid` must
+  equal its own pid; if it does not, stop and relaunch, never fall back to a
+  foreground run. The wrapper also re-sessions itself and ignores SIGHUP, so a
+  tool deadline cannot end it, but it still must not be started in the
+  foreground, where the tool's output pipe is its stdout.
+- Then POLL, at most one check per tool call and under 20 seconds each
+  including any sleep: `kill -0 "$(cat "$pidfile")"` says live; once it is
+  gone, read the newest `$IMX_LOG_DIR/tests/outcomes/wrap-*.json` receipt (it is
+  written by the wrapper at exit) and `tail -n 40 "$log"`. The receipt's
+  `rc`, `timed_out`, `watchdog_triggered` are the outcome; `signal` and
+  `signal_origin` say whether a signal ended the child and who sent it.
+- A receipt with `signal_origin: "external"` (rc -2/-15/-9, `timed_out` and
+  `watchdog_triggered` both false, typically ~30 s after start) means a
+  supervisor OUTSIDE this process tree killed the test — on this host, the
+  GPU reservation guard, because the GPUs in `CUDA_VISIBLE_DEVICES` were not
+  reserved for the wrapper's process tree. That is not a test failure and not
+  your code: record it in debug memory and report the module incomplete with
+  the receipt path; do not retry in a loop and do not change the test scope.
 - Guard optional glob or command-substitution results before passing them to
   stdin-reading commands (`tail`, `head`, `cat`, `grep`); an empty expansion
   turns `tail -n 5 "$log"` into a bare stdin reader that holds the bridge.

doc/RUNBOOK-rebase.md

@@ -69,6 +69,30 @@ requires a fresh module signal, no typed session/audit refusal, and the required
 plan decision from that module's current attempt. Plain completion prose does
 not replace these checks.
 
+### Stopping a run (SIGINT/SIGTERM) and what it leaves behind
+
+SIGINT (Ctrl-C, or a GPU scheduler's `--timeout`) and SIGTERM are a graceful
+shutdown, also when the launcher started the copilot with `nohup … &` and the
+process inherited an ignored SIGINT. The engine stops the running harness
+session (the vendor CLI and its MCP bridge), kills this run's marked background
+processes (including detached test wrappers), releases the GPU lock through the
+run finalizers, keeps the upstream scratch, and exits with `128 + signal`
+(130 for SIGINT, 143 for SIGTERM) within about a minute; a second signal does
+not shorten that. The in-flight module is recorded as `interrupted` in
+`substate.json` (never left `running` or promoted to `done`), `run_status.json`
+says `interrupted`, and `run_trace.jsonl` carries `module_interrupted` and
+`run_interrupted` events. `--resume <run-id>` re-runs interrupted modules and
+skips completed ones.
+
+Module test receipts under `<run_dir>/tests/outcomes/wrap-*.json` name the
+signal that ended a test and its origin. `signal_origin: "external"` (rc
+-2/-15/-9 with `timed_out` and `watchdog_triggered` both false) means a
+supervisor outside the run — on a shared host, the GPU reservation guard —
+killed the test process: check that the reservation covers the wrapper's
+process tree before reading it as a test failure. The bridge's own lifecycle
+log, `<run_dir>/bridge/<step>.lifecycle.jsonl`, records why a tool server
+exited; a `bridge_unavailable` trace event summarises it per attempt.
+
 ## External checkout pin (EXT1-class guard)
 
 Before cutover, the EXTERNAL orchestrator's canonical checkout gains a

doc/architecture/SPEC/app/core.md

@@ -1,6 +1,6 @@
 # app/core.py —— 规范
 
-<!-- verified-against: 2026-10-27 -->
+<!-- verified-against: 2026-10-28 -->
 
 `Copilot` is the headless workflow application. It resolves a TaskSpec,
 delegates reviewer verdict policy to `review.plan_gate`, confirms when needed,
@@ -39,3 +39,11 @@ run（`_contained_run_dir` 拒绝越界路径）；`--repo/--playbook/--task-par
 在恢复时是对已保存 task.json 的**断言**而非修改，任一不匹配即 BLOCKED。
 发布 launcher 因此能把 resume 绑定到不可变的 campaign，而不会误入更新的
 无关任务。损坏的 task.json 同样 BLOCKED 而非崩溃。
+
+## 被信号终止的预约 run（2026-10-28）
+
+`_execute_reserved_locked` 的终态写入（`0 → done`、`BLOCKED_EXIT → blocked`、
+其他 → `failed`）在 `engine.lifecycle.shutdown_requested()` 非空时**跳过**：
+`workflow_execution._interrupted` 已经写下 `interrupted`，`128 + signal` 的
+退出码不得被这里改标成 `failed`（那会让 reconciler 把一个可恢复的 run 当作
+失败收尾）。

doc/architecture/SPEC/app/workflow_execution.md

@@ -1,6 +1,6 @@
 # app/workflow_execution.py — workflow execution boundary
 
-<!-- verified-against: 2026-09-26 -->
+<!-- verified-against: 2026-10-28 -->
 
 `WorkflowExecution` runs an already selected playbook with the caller's
 current LLM dependency. It rechecks the repository context before starting,
@@ -9,6 +9,20 @@ initializes tracing and notification, seeds executor state with checkout and
 adapter policy, and drives `Executor` through `run_guarded`. It returns an
 immutable `ExecutionResult` containing the exit code and blocked reason.
 
+Around `asyncio.run(run_guarded(...))` it installs the run's SIGINT/SIGTERM
+handlers (`engine/lifecycle.py::install_run_signal_handlers`) and restores
+them afterwards. A signalled run ends through `_interrupted`: run_status
+`interrupted` with this process as `child_pid` (`run_status.mark_interrupted`),
+a `run_interrupted` trace event (signal, exit code, completed steps), the
+per-step marks plus `run <id>: interrupted (<SIG>)`, and
+`ExecutionResult(128 + signal, reason)`. Both paths land there: the
+cooperative one (a step saw `shutdown_requested()` and returned blocked, so
+`executor.run` returned normally) and the forced one (the run task was
+cancelled; `CancelledError`/`KeyboardInterrupt` is caught only when a
+shutdown request exists, otherwise it propagates as before). Finalizers have
+already run inside `run_guarded` by then: GPU locks are released and the
+upstream scratch is kept, so the run is resumable with `--resume`.
+
 A losing run lock leaves active run artifacts untouched. A failed knowledge
 lock releases an acquired run lock. A caller-provided lock stays owned by the
 caller. Finalizers and locks run on every executor outcome, including failure.

doc/architecture/SPEC/engine/lifecycle.md

@@ -1,28 +1,59 @@
 # engine/lifecycle.py —— 规范
 
-<!-- verified-against: 2026-08-28 -->
+<!-- verified-against: 2026-10-28 -->
 
-`LOC ~141 · 引擎底座（run 生命周期原语） · refactor-status: ok`
+`LOC ~396 · 引擎底座（run 生命周期原语 + 优雅停机） · refactor-status: ok`
 
 ## 职责
-executor 自己给不了的两条**进程级**保证：同一 run 的互斥锁，与 run 离开事件
-循环时的 finalizer 挂钩点。
+executor 自己给不了的三条**进程级**保证：同一 run 的互斥锁，run 离开事件
+循环时的 finalizer 挂钩点，以及 SIGINT/SIGTERM 到达时**有界的优雅停机**
+（run-20261027-193251：引擎从 `nohup … &` 启动器继承了 SIGINT 的 `SIG_IGN`，
+GPU 调度器 `--timeout` 发出的 SIGINT 被整个忽略，随后 reservation guard 逐个
+杀掉其下的测试进程）。
 
 ## 功能
 `RunLock` 对 `<run_dir>/.lock` 取排他 advisory `flock`（第二个并发
 `--resume` fail fast，而不是交错写进度、共享 checkout）；
 `register_finalizer`/`finalize` 维护逐 run 的异步 finalizer 表；
 `run_guarded` 包住 executor 协程，在完成/失败/抛异常的**每条**退出路径上、
-同一事件循环内执行 finalize。
+同一事件循环内执行 finalize，并把自己的 task 绑定给停机路径
+（`bind_run_task`），退出时解绑。
+
+停机：`install_run_signal_handlers(run_dir, run_id) -> restorer` 为 SIGINT/
+SIGTERM 安装处理器（**无条件** `signal.signal`——继承来的 `SIG_IGN` 正是要
+覆盖的状态；`asyncio.Runner` 只在默认处理器仍在时才装自己的，所以本处理器
+在 `asyncio.run` 内保持权威）；处理器调用 `request_shutdown(signum, …)`
+（幂等：第一次接受，之后只计 `repeats`），它启动守护线程 `imx-shutdown`
+并对绑定的 run task 调 `cancel`。工作线程按序：`terminate_children`
+（对所有已注册 harness 子进程 `kill_tree`，TERM→宽限→KILL，含其 MCP bridge
+后代）→ `cleanup_run_processes(run_dir, run_id)`（带 `IMX_RUN_ID`+`IMX_LOG_DIR`
+标记的后台链，含 detached 测试链）→ 注册的 `shutdown hooks`（模块循环用来
+把在飞模块标 `interrupted`）→ 等主线程正常退出事件循环；超过
+`SHUTDOWN_HARD_DEADLINE_S`（60 s）则再杀一次子进程并 `exit_fn(128+signum)`
+（默认 `os._exit`）。
 
 ## 公开契约
 `RunLock(run_dir).acquire()/release()`（context manager）；`RunLockHeld`；
 `register_finalizer(run_dir, fn)`（`fn(outcome)`，outcome 是 RunOutcome，或
 executor 在产出之前就抛时为 None）；`finalize(run_dir, outcome)`；
 `run_guarded(run, run_dir) -> outcome`；`require_file_locking()` 与
 `FileLockingUnavailable`（POSIX advisory 锁不可用时抛出的 fail-closed 门）。
-调用方：`cli/copilot.py` 在 run/resume 全程持锁并用 `run_guarded` 包
-`executor.run`；finalizer 注册者是 rebase 流水线（flock 释放、scratch 清理、
+停机面：`ShutdownRequest(signum, run_dir, run_id, requested_at, repeats)`
+（`signal_name`、`exit_code = 128 + signum`：SIGINT→130、SIGTERM→143）；
+`shutdown_requested() -> ShutdownRequest | None`（协作式轮询点）；
+`register_child(proc)/unregister_child(proc)/registered_children()`
+（transport 在一次 harness 调用期间注册 Popen，见 `providers/base.py::
+run_harness_cli`）；`register_shutdown_hook(fn)/unregister_shutdown_hook(fn)`
+（同步、幂等、不得抛）；`terminate_children(grace_s)`；
+`request_shutdown(signum, *, run_dir, run_id, hard_deadline_s, grace_s, exit_fn)`；
+`install_run_signal_handlers(run_dir, run_id, *, signals) -> restorer`
+（仅主线程；其他线程返回 no-op restorer）；`_reset_shutdown_state_for_tests()`。
+调用方：`app/workflow_execution.py` 在 run/resume 全程持锁、在
+`asyncio.run` 外侧安装/恢复信号处理器并用 `run_guarded` 包 `executor.run`，
+捕获 `CancelledError`/`KeyboardInterrupt` 后若 `shutdown_requested()` 非空则走
+`_interrupted` 终局协议（run_status `interrupted`、`run_interrupted` trace、
+退出码 128+signum）；`rebase_engine/module_rebase.py` 在每次 attempt 后轮询
+`shutdown_requested()` 并注册在飞钩子；finalizer 注册者是 rebase 流水线（flock 释放、scratch 清理、
 终局报告、CI abort 清理）与 `engine/worktrees.py` 的共享持有释放；
 `require_file_locking` 由 `engine/worktrees.py::materialize` 和
 `idempotency.py`（`key_lock`/`reap_stale`）消费。
@@ -47,24 +78,45 @@ executor 在产出之前就抛时为 None）；`finalize(run_dir, outcome)`；
   finalize 真正完成，然后才让取消向外传播 —— 光 shield 会让外层先抛、
   loop 关闭时把孤儿 finalizer 拦腰取消。
 - 什么都没注册时整个挂钩点是 no-op（现有非 rebase playbook 行为不变）。
+- 停机**恰好一次**：第二个信号（任一种）只计 `repeats`，不重启序列、不改
+  退出码；信号处理器只做"记状态 + 起线程 + `call_soon_threadsafe(cancel)`"，
+  不在处理器内做 I/O 等待。
+- 协作式与强制式两条路径留下**同一份**持久痕迹：模块 `interrupted`、run
+  `interrupted`——前者由步骤自己在看到请求后返回 BLOCKED，后者由取消穿过
+  finalizer（GPU 锁释放、scratch 保留）后在 `workflow_execution` 收口。
+- 硬截止是兜底不是常态：主线程正常退出事件循环（`_run_loop is None`）时
+  工作线程直接返回，由主线程写终局并返回退出码；只有卡死的 executor 才会
+  被 `os._exit(128+signum)` 终结。
+- 子进程注册面向 **cursor-agent/claude/codex 整棵树**：`subprocess.run` 隐藏
+  Popen，所以 transport 必须经 `run_harness_cli` 注册；`terminate_children`
+  用 `kill_tree` 走后代，bridge 不会被 reparent 到 init 后继续持有 run 的
+  工具作用域。
 
 ## 边界 —— 不属于这里
 不含 teardown 的内容本身（各注册方自带）；不做进度/checkpoint 持久化
 （executor）；不是 run 状态查询（`run_status.py`）；不含 step 逻辑。
 
 ## 依赖（允许）
-仅 stdlib（asyncio/os/pathlib/fcntl-守卫）。零 engine 内 import。
+stdlib（asyncio/os/signal/subprocess/threading/pathlib/fcntl-守卫）；停机
+工作线程**延迟导入** `testing/process_tree.py::kill_tree` 与
+`rebase_engine/processes.py::cleanup_run_processes`。零 engine 内 import。
 
 ## 扩展点
 新的 run 级资源清理 = 调用方去 `register_finalizer`，不改这里；只有新的
 **进程级** run 保证才落在这里。
 
 ## 测试
 `test_run_lifecycle.py`（同进程/跨进程锁争用、release 后可重取、无 fcntl
-降级、finalizer 恰好一次 / 异常与取消隔离 / 失败路径也执行）。
+降级、finalizer 恰好一次 / 异常与取消隔离 / 失败路径也执行）；
+`test_run_shutdown.py`（覆盖继承的 SIG_IGN 并恢复、请求幂等与 128+signum、
+工作线程杀注册子进程整棵树后在硬截止 `exit_fn`、无绑定 task 时跑钩子即返回、
+`run_guarded` 绑定/解绑与取消仍 finalize、`run_harness_cli` 注册/超时杀树、
+模块协作式/取消式 `interrupted`、步骤 BLOCKED、wave gate、resume 重跑）。
 
 ## 重构备注
 小而承重。`_finalizers` 是模块级进程状态 —— pop-before-run 就是它的自清
 机制；若将来单进程要并行多 run，把表挂到 run 对象而不是模块全局。
 `run_guarded` 的 shield 循环微妙（取消期间的 teardown 存活是评审加固过的）
-—— 改动前先重跑取消路径的测试。
+—— 改动前先重跑取消路径的测试。停机状态（`_shutdown`/`_children`/
+`_shutdown_hooks`/`_run_loop`）同样是模块级进程状态，与"一进程一 run"的
+前提绑定；测试经 `_reset_shutdown_state_for_tests()` 清场。

doc/architecture/SPEC/engine/steps/rebase_v3.md

@@ -1,6 +1,6 @@
 # engine/steps/rebase_v3.py —— 规范
 
-<!-- verified-against: 2026-10-27 -->
+<!-- verified-against: 2026-10-28 -->
 
 `LOC ~2204 · step 库（v3 rebase 装配层） · refactor-status: oversized`
 
@@ -16,8 +16,8 @@ step —— 薄的受治理 wrapper，substate-first、类型化失败、发布
 | `rebase.v3_scan` | deterministic/read | `manifest_jobs`；`test_manifest.json` 产物 |
 | `rebase.v3_wheel` | deterministic/write_workspace | `upstream_commit`（选 commit → 装进目标 venv → **最后**才 pin Dockerfile：装失败绝不留脏树） |
 | `rebase.v3_assign` | deterministic/read | `active_modules`、`wave1_modules`、`wave2_modules`（path-sync 后按 wave 分派） |
-| `rebase.v3_wave_gate` | deterministic/read | wave-1 失败 ⇒ `wave2_modules=[]`；`halt_on_module_failure` ⇒ ESCALATE |
-| `rebase.v3_module_rebase` | script/write_workspace | `module_<name>_status`（foreach；同 run 串行锁 —— 同一 checkout；substate done/skipped 短路防重入二次执行） |
+| `rebase.v3_wave_gate` | deterministic/read | wave-1 失败**或 interrupted** ⇒ `wave2_modules=[]`；`halt_on_module_failure` ⇒ ESCALATE |
+| `rebase.v3_module_rebase` | script/write_workspace | `module_<name>_status`（foreach；同 run 串行锁 —— 同一 checkout；substate done/skipped 短路防重入二次执行；outcome `interrupted` ⇒ BLOCKED，可 `--resume`） |
 | `rebase.v3_test_loop` | script/write_workspace | `phase3_failed`；substate `tests.pipeline`/`infra_failures`；空 manifest ⇒ `manifest_empty` |
 | `rebase.v3_precommit` | script/write_workspace | 无发布 —— substate `tests.precommit`（passed/failed/failed_preexisting/not_declared） |
 | `rebase.v3_push_gate` | deterministic/read | `push_gate_flagged`、`push_gate_overrides`；不许 ⇒ FORBIDDEN；override 记 trace |
@@ -133,3 +133,16 @@ harness 会话通过 MCP tool bridge 拿到同一套工具面，plan gate 在派
 - baseline venv（`ensure_baseline_venv`）：硬链接克隆 campaign venv 并换入
   baseline commit 的 wheel，让"omni main 对上次对齐的 wheel 是否也失败"可问；
   任何失败返回 None，保留保守的回归判定。
+
+## 信号中断的模块：BLOCKED 收口、resume 重跑（2026-10-28）
+
+`_v3_module_rebase` 在 `rebase_module` 返回 `status == "interrupted"`
+（引擎收到 SIGINT/SIGTERM，见 `engine/lifecycle` 与 `rebase_engine` 规范）时
+返回 `StepResult(False, BLOCKED, "<m>: interrupted by <SIG> — resumable with
+--resume", state_updates={module_<m>_status: interrupted})`：executor 不再推进到
+下一个 foreach 兄弟，run 由 `workflow_execution` 以 `interrupted` 收口。短路
+条件**不变**（只有 done/skipped），因此 `--resume` 重跑 interrupted 模块、跳过
+done；`_v3_wave_gate` 把 interrupted 与 failed 同等视为"wave-1 未完成"（半
+rebase 的模块不是 wave 2 的基础）。测试：`test_assembly.py::
+test_v3_module_rebase_reruns_interrupted_and_blocks_on_interrupt`、
+`test_run_shutdown.py::test_wave_gate_treats_interrupted_as_not_done`。