fix(noema): reject a completion the provider declares truncated

[seonghobae]

Lane: jan | session:session_01BV96rXhqoR3tYZ9AeAVur4

The hole

scripts/ci/noema_review_gate.py never reads finish_reason — the field where an OpenAI-compatible provider declares it stopped at the output-token budget rather than because the model finished. On main@f25063882, grep -c finish_reason returns 0.

Truncation is only sometimes caught downstream. Driving main's own parser directly:

'{"decision":"approve","summary":"looks fine","findings":[{"severity":"high"'
   → NoemaModelOutputError                              ✅ unbalanced, fails closed

'{"decision":"approve","summary":"reviewed","findings":[]}'
   → parsed as a valid verdict                          ⚠️ cut lands on a closed object

The second case is not a corner case. findings is emitted last, so the likeliest parseable truncation is an approval carrying an empty or short findings list — a review cut off mid-thought, arriving as a genuine-looking APPROVE on a required gate, with nothing anywhere saying it was incomplete.

main's local repair is deliberately lossless (_strip_trailing_commas_outside_strings only), so it does not manufacture this. The provider does, and the gate simply never looked at the field where the provider says so.

This is the same failure family as #1921: a reviewer that could not see everything returns APPROVE, silently. There the cause is allowed-locations starvation; here it is output-budget truncation.

The fix

reject_truncated_completion(raw) reads choices[0].finish_reason on the decoded body and raises NoemaModelOutputError for the unambiguous "length". It runs in call_llm immediately after decode_llm_response_body, ahead of content extraction.

Deliberately narrow, in three ways:

• Only "length" is rejected. Missing, empty, "stop", "tool_calls", "end_turn", "content_filter" — all pass untouched. A provider reporting a vocabulary this gate does not model must not be failed spuriously, and this gate fronts every repository in the organization.
• It never raises a shape error. Malformed envelopes keep being classified by extract_llm_message_content, which reports their actual cause with precise messages. Duplicating that here would attribute the wrong reason to a body this function cannot classify.
• No retry, no new exception type, no change to repair ownership. The gateway keeps that, per the caller attempts=1 (gateway owns repair/failover) contract already on main.

Relationship to #1606

This is the portable half of #1606. That PR cannot merge as written: alongside this check it adds caller-side retry (is_retry, truncated_after_retry, a stale-head check before retry) built on extract_llm_completion, which main replaced with extract_llm_message_content when it moved repair ownership to the gateway. Resolving its conflict would mean silently reversing that architectural decision. Detail in #1606.

So the defect is landed here in main's own idiom and its own error type, and #1606's remaining parts — the bounded-field validators and NOEMA_MAX_* constants — stay available as their own change if wanted.

Verification

• pytest tests/test_noema_review_gate.py -k "truncated_completion or call_llm_consults" → 15 passed
• coverage run -m pytest tests -q → 2905 passed, 1 skipped, 21 subtests passed
• coverage report → scripts/ci/noema_review_gate.py 100% (864 statements / 388 branches, 0 missing); TOTAL 100%
• interrogate → PASSED (100.0%)
• git diff --check clean
• Branched from origin/main@f25063882 in a fresh worktree

The first test asserts the truncated payload really would have parsed before the guard — otherwise the test would pass for the wrong reason and prove nothing about the hole being closed. A separate test pins that the guard runs ahead of content extraction, so a future reorder cannot silently move it after the parse.

Developer experience

A truncated review now fails with a message naming the cause (finish_reason=length) instead of either succeeding wrongly or failing later with a misleading JSON-shape error. The message embeds no part of the untrusted response body, so it stays safe in the public pull_request_target job log.

User experience

An APPROVE from noema-review can no longer be produced by a completion the provider itself declared incomplete. This closes a path by which a required gate could pass a change nobody finished reviewing.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4

---

Generated by Claude Code

[coderabbitai]

Warning

Review limit reached

Next included review available in 12 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Team

Run ID: 88d2a730-71d3-46a6-8f65-062fc4e0a99d

📥 Commits

Reviewing files that changed from the base of the PR and between efb8926 and 9161e50.

📒 Files selected for processing (2)

• scripts/ci/noema_review_gate.py
• tests/test_noema_review_gate.py

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[seonghobae]

CI status note — standing down on the two red checks on head 88588b31b.

CodeQL compatibility analysis (python) and (actions) ended with VERDICT_STATE=pending: the compat job's designed fail-fast after dispatching a CodeQL PR scan whose terminal verdict has not been published yet. The dispatched scan (run 33977184168, re-dispatched 16:13Z by the ready-for-review event) is queued behind the organization job ceiling — of 70 CodeQL PR runs in .github since 13:00Z, 44 are queued, 20 cancelled, 2 failed, 0 succeeded. Nothing in this PR's diff is involved, and the dispatch workflow reruns this exact job once the verdict lands.

Not re-running it manually: a re-run dispatches again and enlarges the queue that is causing the failure (docs/doctoring/ci-failure-signature-triage.md §9). Capacity relief is in #1938.

---

Generated by Claude Code

[seonghobae]

CI note and push: head 88588b31b → 773e70139 (a base merge only; no change to the PR's own commits).

Two new red checks on the previous head, both the single-upstream gateway stall class and neither in this diff: strix (job 101353895177) ended "provider/backend was unavailable", and noema-review (job 101354120806) failed with the gateway 502 after 1685 s. #1939 removed that stall on main at 17:25Z, but a re-run cannot pick it up because workflow_sha is bound at run creation (catalog §2), so this push merges main@f2f91b806 into the branch to create the new event. The two CodeQL compatibility failures remain the designed pending state noted above.

Local gate on 773e70139: 2911 passed / 1 skipped, coverage 100% (13144 statements, 5312 branches; scripts/ci/noema_review_gate.py unchanged at 100%), interrogate 100%, git diff --check clean. Cadence: 9 h 19 m since the previous push.

---

Generated by Claude Code

[seonghobae]

CI note on head 773e70139: noema-review (run 33992904601, job 101388218941) failed in "Prepare Noema model verdict" with HTTP Error 429: Too Many Requests; caller attempts=1, duration=1444.7s, phase=response_error, served_model=deepseek-ai/deepseek-v4-pro-0813. Gateway-side; nothing in this diff (noema_review_gate.py's truncated-completion rejection and its tests) is on that path — the request never returned a completion to reject.

Same class as #1938's failure 20 minutes earlier, read per host 1's source trace on that thread: the gateway did fail over; served_model names the last route tried; 1444.7 / 548 ≈ 2.6 says two or three of the ready routes each waited out their per-recv timeouts before answering 429. The run was created at 21:23Z, so its sidecar is post-#1939 (diversified catalog) but its workflow predates #1944, so there is no noema-sidecar-evidence artifact to show the per-route walk.

Corrected tally, since the one I posted on #1938 was wrong: of the post-#1939 noema-review runs in this repository that actually reached the verdict step, 1 succeeded (#1902) and 3 failed (#1872 502 after 1989.9 s, #1938 429 after 551 s, this one 429 after 1444.7 s). The three run-level "successes" at 21:59–22:15Z were the closure-event runs of #1943/#1944/#1945 after merge, whose job was skipped entirely.

Action: not re-running yet. #1938's attempt-2 re-run (queued 23:49Z, same pre-#1944 workflow) is the measurement of whether the rate has moved; at 1 in 4 a re-run here would spend another ~25 minutes of a runner slot in a 234-deep queue at the same odds. I will issue this PR's single re-run once that result is in, or fold it into the next base-merge push, which also binds the artifact-producing workflow. The two CodeQL compatibility failures remain the designed pending state noted earlier.

---

Generated by Claude Code

[seonghobae]

CI note on head 773e70139, second failing check: strix (run 33992904674, job 101388406980) ended "provider/backend was unavailable" after 23.5 minutes in "Run Strix (quick)". Gateway-side, not this diff — and this one carries the evidence the Noema failure above could not, because the Strix workflow already ships the sidecar files in strix-reports (artifact 9979385164):

• contextual-orchestrator-preflight.json: ready 1 / rejected 11 of the 12 selected routes. All four openrouter routes 429; nvidia_nim deepseek-v4-pro-0813 and deepseek-v4-flash-0731 429; nvidia_nim_sub deepseek-v4-flash-0731 TimeoutError; the four gemma-3 routes 404 on both NVIDIA accounts. The only ready route was nvidia_nim_sub deepseek-v4-pro-0813.
• strix.log: that sole route answered 429 rate_limit_exceeded on the first request and on all five of Strix's replays (backoff 2 → 32 s, 00:06–00:12Z); run.json shows llm_usage.requests: 0, so the scan never got a single completion.
• contextual-orchestrator-sidecar.stderr.log: a wall of request_failed status=429 code=rate_limit_exceeded (the pre-fix(sidecar): let the stream sanitizer pass orchestrator route and circuit events #1945 sanitizer, so no per-route lines).

So at 23:48–00:12Z the free pool was rate-limited on every account at once — OpenRouter at preflight, the primary NVIDIA account at preflight, the sub account on request. #1939's interleave is working (4+4+4 selected); there was simply no free capacity to interleave. The same window produced #1938's and this PR's noema-review 429s, and it burned roughly 25 minutes of a runner per job in a 234-deep queue while doing so.

Action: no re-run now — with one ready route that 429s on contact, a re-run is 25 more minutes of slot at near-zero odds. This PR's single sanctioned re-run stays gated on #1938's queued attempt 2 (or the next base-merge push), same as the Noema check. The lever for the condition itself (a pool with ≤1 ready route that rate-limits on first contact should not cost 25 minutes) sits in the orchestrator's retry policy and the Strix replay policy, neither in this diff; I have handed the artifact to the observability lane.

---

Generated by Claude Code

[seonghobae]

CI note on head 773e70139, third failing check: opencode-review (run 33992904744, job 101394230478) failed in seconds at "Fail closed without a current-head OpenCode verdict" — catalog signature 1, not a review verdict; no opencode-agent review exists on this head. The dispatch step succeeded and created handler run 34002410471 (OpenCode Review Dispatch, queued 00:52Z), but every handler run completed since 2026-09-05T17:00Z (83 of 83) was rejected in validate-pr-metadata because the dispatcher identity opencode-agent[bot] is not in OPENCODE_REPOSITORY_DISPATCH_ACTOR (still github-actions[bot]). #1932's multi-identity parser is on main; the variable is an owner action tracked in #1929.

Nothing in this diff is involved, and no push or re-run from this side changes the outcome — a re-run re-dispatches into the same rejected gate, and the dispatch workflow re-runs this exact job by itself once a verdict is published. Standing down on this check until #1929's variable is reconciled; the Strix, Noema, and CodeQL notes above stand.

---

Generated by Claude Code

[seonghobae]

CI note and push: head 773e70139 → 9161e50b2 (a base merge only; no change to the PR's own commits; 5 h 53 m since the previous push).

Merges main@efb892692, the owner's bypass-merged sidecar pin bump to contextual-orchestrator@414f2297 (contextual-orchestrator#1081: _invoke's retry-then-failover no longer stacks on the client's max_retries — the loop behind this head's noema-review 429 after 1444.7 s and its 2 h 18 min Strix), plus #1943 / #1944 / #1945. A re-run could not pick any of that up (workflow_sha and the sidecar pin are bound at run creation), so this push is the remedy for this head's failed noema-review and strix.

Local gate on 9161e50b2 with GITHUB_ACTIONS=true: 2919 passed / 1 skipped, coverage 100% (scripts/ci/noema_review_gate.py unchanged at 100%), interrogate 100%, git diff --check clean. The pool condition (#1948) is unchanged by the bump, so a Noema or Strix failure on this head is still possible; if it happens, the run now ships its per-candidate trace as an artifact. opencode-review stays fail-closed until #1929's variable is set; the CodeQL shards will read designed-pending again.

---

Generated by Claude Code