Monitoring: run the four-arm correction-timing pilot and apply the stop gate

[w4ffl35]

Current status (2026-09-30)

The scripted four-arm local smoke reported 100% completion in every arm and a STOP for that scripted run. It used target-aware scripted operations, synthetic token increments, and one corrected flag for both correction messages, so it does not answer this issue's causal question for a coding model. The later DeepSeek preflight was baseline-only on training fixtures; it did not run the frozen M/R/G arms. No qualifying predictor or complete held-out four-arm model result is posted. Keep this issue open, dependent on #11's frozen artifacts, and preserve the original randomization, accounting, and stop gate. The current RSI proof #15–#20 is separate from learned-monitor expansion.

Original pilot specification (historical context)

The design below records the original requested scope and acceptance checks. The current status above identifies what remains to be completed.

What problem are you trying to solve?

A predictor can identify stalled trajectories without improving an agent. Conversely, any interruption may improve performance by causing another reasoning pass. The held-out evaluation must separate learned timing from generic reflection, message content, extra compute, and the existing deterministic guardrails.

This issue depends on #10 and #11. Run it only if #11 freezes a qualifying predictor without inspecting held-out outcomes.

Proposed behavior

Run the eight held-out fixtures from #11 with two repetitions and four randomized arms, for 64 scheduled test runs:

Arm	Experimental correction
B	None; current deterministic guardrails only
M	Frozen learned threshold timing with the fixed specific correction
R	Independently sampled timing with the same specific correction
G	Learned threshold timing with a fixed generic reflection message

Freeze these messages before the test:

• Specific: “Check the latest tool result. Re-read the affected configuration if necessary, identify the failed precondition, and change the next action before retrying.”
• Generic: “Pause and review your approach before continuing.”

Derive R's intervention probability and boundary-index distribution from calibration-set M triggers. Record non-delivery when a scheduled boundary is not reached. Permit at most one experimental correction per run. Every arm retains identical deterministic guardrails and the same model, sampling settings, call limit, token limit, time limit, and tool surface.

Budget

The complete development plus held-out pilot has a mathematically consistent ceiling:

• 32 development runs from Monitoring: build the restricted stall-prediction pilot and frozen baseline #11 plus 64 held-out runs here = 96 scheduled runs.
• Per run: at most 16 model calls, 20,000 total input-plus-output tokens, and five minutes.
• Global ceiling across both issues: 2,000,000 tokens or 10 inference hours, whichever occurs first.
• Admit a run only when its full remaining allowance fits the global budget.
• Retries and aborted attempts consume the same global budget.
• If the full held-out schedule cannot complete, report an incomplete pilot that cannot pass the expansion gate.

Report success-versus-token checkpoints at 5,000, 10,000, and 20,000 tokens, counting unfinished runs as unsuccessful at each checkpoint.

Measurements

Keep raw values and denominators. Do not use a weighted composite score.

• Verified success and valid-run rate.
• Predictor precision, recall, AUCPR, and Brier score on untouched B trajectories.
• Milestones, redundant calls, invalid retries, forbidden-effect attempts, correction delivery, and existing guardrail interventions.
• Paired B-success/M-failure harm rate.
• Total tokens, calls, wall time, predictor p50/p95 latency, and peak monitor memory.
• Energy remains “unmeasured” unless it is actually instrumented.

Also present a 0.0–1.0 comparison grid for usefulness, detection, safety, harm avoidance, token efficiency, time efficiency, and monitor efficiency. Efficiency must not compensate for task failure.

Analyze paired fixture/repetition outcomes and task-cluster bootstrap intervals. Fixtures, rather than individual trajectory windows, are the independent units. Headline results use intention-to-treat arm assignments and never condition on correction delivery.

Go/stop decision

Expand only to one bounded real-repository transfer study if all conditions hold:

• M exceeds B verified success by at least 10 percentage points.
• M exceeds R and G by at least 5 percentage points each.
• The paired 90% task-cluster bootstrap lower bound for M−B is above zero.
• B-trajectory prediction precision is at least 0.80.
• Harmed-pair rate is at most 0.10.
• Mean M token use is at most 1.10× B.
• Predictor p95 latency is below 10 ms.
• There are zero realized forbidden effects and no evaluator, split, or journal integrity failures.

If the test is inconclusive or fails an efficacy gate, stop learned-monitor expansion and retain the deterministic guardrails. One bounded rerun is allowed only for a documented instrumentation or fixture defect and must use fresh held-out fixtures. Do not tune against held-out outcomes.

Alternative approaches

• Compare only B and M. Rejected because it cannot distinguish learned timing from the effect of an extra reflection turn.
• Add trees, recurrent networks, LLM monitors, or SNNs to the first matrix. Rejected because the first decision is whether any learned correction timing has causal value.
• Resume autonomous RSI after a positive synthetic result. Rejected; at most, the result earns one separate transfer study.

Security / least-privilege impact

Run only through the restricted simulator from #11. The monitor remains advisory and cannot change tools, permissions, budgets, evaluator state, or fixture state. This experiment establishes neither an OS sandbox nor safe broad authority.

Acceptance criteria

• Arm assignments, fixtures, predictor, thresholds, templates, model digest, analysis code, and budget manifest are frozen before the first held-out run.
• All scheduled, aborted, invalid, and retried attempts appear in the final accounting.
• Detection metrics use untouched B trajectories; intervention-altered windows are not relabeled as false alarms.
• Raw results, normalized grid, paired uncertainty, realized correction dose, budget use, and every go/stop gate are reported.
• The result is posted to this issue and preserved in a machine-readable local artifact with hashes.
• No new learner or RSI work is opened automatically. Any continuation requires a new decision based on this result.

Additional context

This is the decision-producing issue. It should end with a clear STOP or ONE TRANSFER STUDY verdict, including the strongest limitation on that conclusion.

[w4ffl35]

Tracked by #13. This is the decision-producing held-out evaluation and must use the artifacts frozen by #11.

[w4ffl35]

Review of local pilot result

The local report for monitor-pilot-20260922185924 records 96 scheduled runs, B/M/R/G success of 100%/100%/100%/100%, M−B of 0 percentage points, a paired lower bound of 0, and a STOP verdict. No transfer study or RSI expansion follows from this run.

The result has a narrower meaning than the planned causal evaluation:

• run.ts uses scripted-restricted-model@sha256:monitor-pilot-v1, not an external coding model. The report sums run durations to 29 ms. The reported 864,000 tokens are accounting increments of 1,000 input and 200 output tokens per scripted operation, not measured model usage.
• simulator.ts passes the fixture, including targetFiles, directly to scriptedOperation. The scripted policy can apply the oracle target without inferring it from the visible task and tool results.
• The specific and generic correction templates are appended to a message array, but the scripted policy responds to the same corrected boolean for both. This implementation cannot measure an effect of correction wording.
• The evaluation calls runFixture directly. It does not exercise the HeadlessSession iteration-boundary hook added under Monitoring: complete iteration-boundary hook acceptance checks #10.
• Arm B's reported summary combines 32 development runs with 16 held-out runs; M, R, and G each contain only 16 held-out runs. Arm comparisons should use the same held-out set.

These facts make this a useful restricted-simulator and journal smoke test. They do not establish whether learned correction timing improves, harms, or fails to help an actual coding model. The 100% scripted baseline creates a ceiling, so this run cannot pass the predefined improvement gate. Record STOP for this run, keep learned-monitor expansion paused, and do not treat it as the decision-producing model evaluation specified in this issue.

The report and journal are local under .headlesscode/monitor-pilot/monitor-pilot-20260922185924/; the current implementation and architecture document are uncommitted in the working tree. No new run is proposed by this review.

[w4ffl35]

Baseline-only real-model preflight

The earlier monitor-pilot-20260922185924 STOP applies to its scripted run only. This preflight used HeadlessSession with the actual iteration-boundary hook and a no-op monitor. The model saw the task, fixed system instructions, its own messages, and results from five restricted virtual tools; it did not receive fixture target files, oracle state, or stall labels. Only the 12 training fixtures were run. Calibration and held-out outcomes were not used.

The protocol was frozen before model requests: deepseek/deepseek-v4.1-flash-20260910 through DeepSeek's OpenRouter endpoint, temperature 0, at most 384 output tokens per call, 12 calls and 7,900 measured tokens per run, five minutes per run, and 100,000 measured tokens globally. The task order, three-turn stall label with short terminal windows masked, and the two-part feasibility gate are in .headlesscode/monitor-preflight/preflight-20260922193300/manifest.json.

The replay-verified result is 7/12 independently verified successes and 3 distinct trajectories with a qualifying stall window (5 windows total; 40 terminal windows masked). That meets the frozen feasibility gate of 3–9 successes and at least 3 stalled trajectories. The 12 final task outcomes include 11 valid runs and one invalid run: a virtual execute_command call omitted the required validate token and was blocked with no host effect. That invalid run supplies the third stalled trajectory; only two valid trajectories stalled. This sensitivity is the strongest limitation on the gate result.

Across all attempts, including three earlier aborted attempts, the provider reported 63,952 input + 9,243 output = 73,195 measured tokens over 73 model requests. One HTTP 400 rejection had no provider usage or request ID, so its usage is unavailable and is not included in that measured total. The successful responses' generation IDs, provider request IDs, model identity, per-call usage, raw trajectories, and per-task outcomes are recorded locally in verified-report.json and the referenced journals. Every final virtual trace, tool result, final artifact hash, oracle outcome, and stall label was replayed successfully. The largest final run used 7,144 measured tokens and 6 model calls; all final runs stayed within the wall-time limit.

The first generated final-report.json applied an extra all-valid condition that was absent from the frozen gate and incorrectly said STOP. verified-report.json records that analysis correction and is the verified result. These artifacts and code are currently uncommitted.

Recommendation: The specified feasibility gate passes, so a fresh four-arm evaluation may be warranted under a separate decision. This preflight does not test a learned monitor or show that it improves performance. Do not retrain a predictor or resume RSI work on this result.