A retryable OpenRouter 402 (in_flight_budget_exhausted, Retry-After: 120) is classified as terminal out-of-credit

[thegeorgepu]

Observed live on both worker boxes on 2026-08-29 (worker build 604c428c). Every dispatched task died at its first model call. The tasks were not retried and not backed off — each was burned immediately.

What the provider actually returned

402: {"message":"This request would exceed your available credits given your current
in-flight requests. Retry after in-flight requests settle, or add credits.",
"code":402,"metadata":{"reason":"in_flight_budget_exhausted",
"limit_source":"openrouter_in_flight_budget",
"remedy_hint":"Retry after your in-flight requests settle (see the Retry-After header). ...",
"headers":{"Retry-After":"120"}}}

Note the shape: a Retry-After header, and a remedy that begins "Retry after your in-flight requests settle". OpenRouter is describing a transient concurrency/budget cap, semantically much closer to a 429 than to "your account is empty".

Why it is classified as terminal

vinci/extensions/vinci-provider.ts:24 matches any 402 as budget-exhausted:

/\b402\b|budget[_ -]?exhausted|insufficient[_ -]?quota|out of credits?|.../i

Two independent clauses fire here — the bare \b402\b, and budget[_ -]?exhausted, which matches the literal string in_flight_budget_exhausted. vinci/extensions/vinci-model-provenance.ts:132 likewise returns "account" for status === 402, and vinci/test/no-downgrade-integration.mjs:29 states the assumption outright: "402 is out of credit".

That assumption is right for a genuine out-of-credit 402 and the no-downgrade behaviour built on it is correct — silently falling back to a cheaper model on a billing failure is exactly what should not happen. The defect is that two different 402s are conflated. One is terminal; the other ships a Retry-After and asks to be retried.

Measured impact

run	boxes	dispatched	FAILED	spend
night3	box-1 + box-2	4	2	$0.021
night3ab	box-1 only	1	1	$0.001

Reproduced at single-box, one in-flight request, so this is not only a concurrency artifact — but that also means the honest fix is not "retry forever". With a depleted balance a bounded retry still fails; what changes is that a recoverable dip (in-flight settling, a top-up landing mid-run) stops costing the whole queue.

Boxes were otherwise healthy throughout: build 604c428c on both, vinci-worker-daemon.service active, claim directories and live pids observed.

Ask

Split the classification:

• metadata.reason === "in_flight_budget_exhausted" (or any 402 carrying Retry-After) → retryable; honour the header with bounded backoff and a retry ceiling, then go terminal.
• 402 without those markers → terminal out-of-credit, unchanged, still no downgrade.

vinci/extensions/vinci-provider.ts:153-154 already honours retry-after on the rate-limited path, so the mechanism exists; the 402 path just never reaches it.

Second, smaller observation — reported as unconfirmed

On both failures the task record disagrees with itself: outcome.state = "BLOCKED" while the task-level state = "FAILED" with exit_code: 1. Night 2 recorded BLOCKED outcomes as BLOCKED task states (3 of 72), so the mapping is not unconditionally lossy. I have not read the mapping code and am not asserting a defect — flagging it as worth a look, since a blocked-vs-failed mislabel changes what the endurance exit criteria count.

Repro: dispatch any worker task while the OpenRouter balance is below the in-flight budget threshold; inspect /var/lib/vinci-worker/tasks/<id>.json.

[thegeorgepu]

Follow-up: the second observation in this issue is now confirmed, with a control. I filed it as "reported as unconfirmed — I have not read the mapping code and am not asserting a defect." I still have not read the mapping code, but the behaviour is now reproducible in one direction and contradicted in the other, which is enough to assert the inconsistency itself.

The 402 path loses the BLOCKED classification. Other paths do not.

task	trigger	outcome.state	task state	agrees?
msg_c56ba6ef	OpenRouter 402	BLOCKED	FAILED	no
msg_1f9c47eb	OpenRouter 402	BLOCKED	FAILED	no
msg_2d1f6434	guard: shell-based file write	BLOCKED	BLOCKED	yes
night-2, 3 tasks	(various)	BLOCKED	BLOCKED	yes

So this is not "BLOCKED outcomes are always flattened to FAILED" — the harness-stop path maps BLOCKED through correctly, and does so with a full explained_by chain naming the guard and the remedy. It is specifically the provider-error path that arrives with outcome.state = BLOCKED and is recorded as FAILED, exit_code=1.

Two consequences worth separating:

1. It mislabels the ledger. A task the provider declined is booked as a worker failure. Endurance exit criteria count FAILED, so every intermittent 402 currently reads as a fleet defect rather than an upstream refusal — which is exactly the wrong direction, because the FAILED count is what someone would use to decide the daemon is unhealthy.
2. It compounds the primary bug in this issue. If the 402 were classified retryable (the main ask above), most of these would never reach a terminal state at all. Fixing the classification likely removes most instances of the mapping problem too — so these are one fix and a follow-on, not two independent ones.

Both instances are deepseek/deepseek-v4-flash-0731 on worker build 604c428c, endurance night3b. Records are on box-2 at /var/lib/vinci-worker/tasks/msg_1f9c47eb.json and box-1 at msg_c56ba6ef.json.

Ongoing rate for scale: over 41 dispatches tonight the 402 has cost 2 tasks outright, against 1 no-commit (#28) and 1 legitimate guard block — so roughly 5% of the night is being discarded on a provider error that ships Retry-After: 120.

[thegeorgepu]

A third distinct 402, with a different remedy again — and this one is individually actionable by the client.

msg_0c19a0cf (night3b, box-1, cost_usd: 0):

402: {"message":"This request requires more credits, or fewer max_tokens.
You requested up to 131072 tokens, but can only afford 23014. ..."}

outcome.state = BLOCKED, task state = FAILED, exit 1 — same mapping problem as the other two, and again zero cost because it never reached the model.

We now have three semantically different conditions sharing status 402:

variant	what it means	correct client response
in_flight_budget_exhausted (+ Retry-After: 120)	transient concurrency cap	wait the header, retry
requires more credits, or fewer max_tokens (+ an exact affordable number)	the request as sized cannot be afforded	retry with max_tokens at or under the stated number
true out-of-credit	account empty	terminal, no downgrade — current behaviour, correct

Only the third is terminal. The current code treats all three as the third.

The middle one is the sharpest illustration of the bug, because the provider hands us the fix in the error body. It says the request wants 131072 tokens and 23014 are affordable. A client that re-issued at max_tokens: 23014 would very likely succeed — 23k output tokens is ample for every one of these endurance tasks, which produce a single short markdown file. Instead the task is discarded at zero cost and booked as a worker FAILED.

That also means the \b402\b catch-all in vinci-provider.ts:24 is not merely imprecise, it is discarding a machine-readable remediation. Whatever shape the fix takes, parsing metadata / the affordable-token count is worth more than a boolean retryable/terminal split.

Suggested refinement to the ask above — classify on the body, not the status:

• in_flight_budget_exhausted → retry after Retry-After, bounded.
• affordability message carrying a number → retry ONCE with max_tokens clamped to it; terminal if that also fails.
• anything else 402 → terminal, unchanged.

Operational note: the affordable ceiling (23014 tokens) is a live readout of how little balance is left, and it will keep shrinking. Expect the mix of these three variants to shift toward the third as the balance drains.

[thegeorgepu]

George here, I think we did run out of credits, so treat it as that. I will add more credits to OpenRouter later.

[thegeorgepu]

Rescoping, and correcting a false premise in the original report

The premise correction first, because it sent an implementation down the wrong path.

This issue states:

vinci/extensions/vinci-provider.ts:153-154 already honours retry-after on the rate-limited path, so the mechanism exists; the 402 path just never reaches it.

That is wrong. The retry-after read near there lives inside the device-authorization polling loop — the code that polls while a user authorizes a pairing code in the browser. It is not on the model-call path, and there is no existing retry-after mechanism there to reuse. An implementation attempt went looking for it and correctly reported it does not exist.

I relayed that claim into an implementation spec without checking it. My error, recorded here so the next reader does not inherit it.

What the model-call retry actually does

transient is consumed at vinci-advisor.ts:105 and vinci-council.ts:138:

if (kind === "transient" && attempt < SAME_CLASS_ATTEMPTS) continue;

SAME_CLASS_ATTEMPTS = 2 (vinci-advisor.ts:34), so that is exactly one retry, and the continue has no delay of any kind — not exponential backoff, not a fixed sleep. It is an immediate second attempt inside the same loop.

This is why the obvious fix does not work. Classifying in_flight_budget_exhausted as transient gets it retried microseconds later, against a cap the provider says needs ~120 seconds to settle. Every retry fails identically, and the task is burned anyway — the same identical-failure loop, just twice as slow. A transient label over a retry that provably cannot succeed is a name asserting more than the code does.

Rescoping: two prerequisites, both real features

Making either case genuinely retryable needs infrastructure that does not exist today:

1. A delay mechanism in the shared retry loops. Honour Retry-After (with a ceiling) before a same-class retry. This touches advisor/council/scope/loopbreak, which serve every failure class — so it is deliberately not bugfix scope.
2. A path to carry the affordable token count from the provider error, through classification, to the request layer, so a retry can clamp max_tokens to the number the provider hands back. The classifier has no access to request parameters.

Until both exist, all three 402 causes correctly remain terminal.

What is being fixed separately

A narrower change is in review: the three causes currently produce the same user-facing message telling the user to go buy credits. For a transient in-flight cap the correct remedy is to wait; for an affordability refusal it is to resize the request. That is live, misleading, and fixable without either prerequisite.

This issue stays open for the two prerequisites above.

Note on the no-downgrade guarantee

Unchanged and must remain so: a genuine out-of-credit 402 is terminal with no model downgrade. Silently falling back to a cheaper model on a billing failure is exactly what must not happen, and no part of this rescoping touches that.

[thegeorgepu]

Correction: my rescoping comment above was wrong, and #38 has landed

I argued above that making 402s retryable needed two things that "do not exist" and were "features, not bugfix scope":

1. a delay mechanism in the shared same-class retry loops, and
2. a path to carry the affordable token count to the request layer.

#38 built both, merged 2026-08-29T23:35:54Z. On main now:

• VINCI_BUDGET_ERROR → VINCI_TERMINAL_BUDGET_ERROR, with the bare \b402\b alternation removed — so a 402 is no longer terminal by status alone
• boundedRetryAfterMs() honouring Retry-After with a MAX_RETRY_AFTER_MS = 300_000 ceiling
• MAX_IN_FLIGHT_RETRIES = 3 and MAX_TOTAL_402_RETRIES = 10
• vinciAffordableTokenLimit() parsing the affordable count, validating the canonical two-number form and rejecting malformed grouping

My reasoning was sound up to the last step and then took the wrong turn. It is true that classifying a 402 transient while the retry provably cannot succeed is worse than leaving it terminal — an immediate retry against a 120-second cap fails identically, just twice. But the conclusion should have been "therefore build the delay," not "therefore leave it terminal and call the delay out of scope." I let the absence of a mechanism decide the scope instead of treating the missing mechanism as the work.

A narrower PR I had open (#39, messaging only) is closed as superseded.

What still stands from that comment

The premise correction does hold, and is worth keeping: this issue's original claim that vinci-provider.ts:153-154 already honoured retry-after on the rate-limited path was false — that code is the device-authorization polling loop, not the model path. #38 had to add the model-path handling rather than route to something existing.

One thing worth checking against #38

A review of #39 caught a real ordering hazard in my version: an affordability guard placed before the status checks would fire on any status whose body matched can only afford N, reclassifying a non-402 transient error to terminal and silently stopping its retries. I have not audited #38 for the same hazard. If its affordability detection is not gated on the status code, it is worth a look.

Leaving this issue open or closed to #38's author — I have no basis to judge whether their fix covers everything reported here.