Skip to content

A returned correction trains: an engine still loading refuses before any job exists (dispatch retries), and a pass's return dispatches its bucket below threshold (card d53a9e0e, gate 3) - #5575

Merged
joelteply merged 2 commits into
canaryfrom
d53a9e0e/a-returned-correction-trains-an-engine-s
Oct 28, 2026
Merged

joelteply merged 2 commits into
canaryfrom
d53a9e0e/a-returned-correction-trains-an-engine-s

Conversation

@joelteply

Copy link
Copy Markdown
Contributor

The windows 5090 after b638145 (Codex, 2026-10-28 20:16-20:22Z): #5563 returned Kimi's
canonical jobs, then (1) re-dispatched job c9e5b989 failed in 2 ms on /train/measure
'HTTP 503 Loading model', ended FAILED with no partition and stranded its examples;
(2) her code/owner bucket held 11 returned examples under minExamples 16, never firing.

(1) measure_examples types its failure (MeasureError::NotReady for HTTP 503 or a
refused connection). create_job checks the served-window measure before request.json,
a run or a partition exist and refuses TrainerUnavailable, the trigger's retryable
refusal: the batch stays owned and recovery retries it once the engine serves.
(2) A recovery pass that returns examples (an ended job's parked remainder, an
engine-refused job) dispatches that bucket at once through the flush verb's own
dispatch (dispatch_returned): they met a threshold when first dispatched. A return
happens only when her window can train one or a smaller chunk is left, so no loop.

Tests: fake_lane 'loading' (503): TrainerUnavailable, nothing posted, no job dir;
a returned batch below threshold dispatches once, whole, in order.
Local (enrico, -j2): continuum-dreaming training_trigger 83/0, continuum-genome
engine_lora_adapter 47/0; check --lib clean in both crates.

Co-Authored-By: Claude Opus 5.5 (1M context) noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE

…any job exists (dispatch retries), and a pass's return dispatches its bucket below threshold (card d53a9e0e, gate 3)

The windows 5090 after b638145 (Codex, 2026-10-28 20:16-20:22Z): #5563 returned Kimi's
canonical jobs, then (1) re-dispatched job c9e5b989 failed in 2 ms on /train/measure
'HTTP 503 Loading model', ended FAILED with no partition and stranded its examples;
(2) her code/owner bucket held 11 returned examples under minExamples 16, never firing.

(1) measure_examples types its failure (MeasureError::NotReady for HTTP 503 or a
refused connection). create_job checks the served-window measure before request.json,
a run or a partition exist and refuses TrainerUnavailable, the trigger's retryable
refusal: the batch stays owned and recovery retries it once the engine serves.
(2) A recovery pass that returns examples (an ended job's parked remainder, an
engine-refused job) dispatches that bucket at once through the flush verb's own
dispatch (dispatch_returned): they met a threshold when first dispatched. A return
happens only when her window can train one or a smaller chunk is left, so no loop.

Tests: fake_lane 'loading' (503): TrainerUnavailable, nothing posted, no job dir;
a returned batch below threshold dispatches once, whole, in order.
Local (enrico, -j2): continuum-dreaming training_trigger 83/0, continuum-genome
engine_lora_adapter 47/0; check --lib clean in both crates.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE
@joelteply

Copy link
Copy Markdown
Contributor Author

Review at 5f9bd4e: blocking recovery gap.

[P1] Make returned-batch dispatch eligibility durable. return_parked_remainder commits acceptance/return before remainders.rs calls dispatch_returned. A crash or cancellation between them, or the run_owned error branch, leaves the below-threshold batch accepted without an active dispatch. On restart, AlreadyReturned is ignored; recover_pass calls dispatch_pending only when ready_to_dispatch is true, which still requires the original threshold without an active dispatch. The same stranded-return failure therefore survives this patch across restart.

Persist an idempotent returned-dispatch intent or eligibility with acceptance, and consume it through normal recovery. Avoid indiscriminately flushing historical returns. Add a fixture that restarts after return acceptance but before dispatch, then runs recovery twice and proves one complete batch is dispatched exactly once. The current test calls the helper twice in the same state and misses this boundary.

The early typed TrainerUnavailable handling for HTTP503/connect failure looks appropriate in source review. Owner reports 83 dreaming and47 adapter tests passing; I did not execute those locally. No adoption/learning verdict yet.

…arries 'returned', and recovery dispatches its bucket below threshold, once across restarts (Codex on #5575)

The flush after a return ran after its receipt; a crash between the two left the
examples below threshold with AlreadyReturned, the same strand. Now PendingBatch.returned
is persisted with the submission that accepts a return (submit_batch_inner, returned_job),
kept exact per part through split (submission_returned), and ready_to_recover (the
recovery tick only) dispatches a bucket holding one. Submit keeps its threshold; rows
from before this read false, so no historical bucket is flushed. dispatch_returned is
deleted: one dispatch path.

Fixture: return accepted, core dies before any pass; owner 2's recovery dispatches it
whole, owner 3's dispatches nothing more; a plain submit below threshold stays held.
Mutation (ready_to_recover -> ready_to_dispatch) fails it at owner 0. Local (enrico, -j2):
continuum-dreaming training_trigger 83/0; check --lib clean.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE
@joelteply

Copy link
Copy Markdown
Contributor Author

Source review PASS at 24c01a7; the earlier return-to-dispatch hold is resolved.

Eligibility now persists in the accepted submission, survives hydration, and is consumed by normal recovery without bypassing an in-flight job/trial. Per-submission flags preserve returned eligibility across budget splits. Historical submissions default false. The replacement fixture restarts after acceptance but before dispatch, recovers through two owners, verifies the whole return is sent once, and checks that an ordinary below-threshold submission stays held.

The early 503/connect TrainerUnavailable path still creates no job artifacts, retaining retry ownership in the dispatch. Owner reports 83 trigger and47 adapter tests passing; displayed Windows check and hygiene pass, remaining CI is in progress. This is source approval, conditional on required checks. No live adoption or learning-gate pass claimed.

@joelteply
joelteply merged commit 4f1b548 into canary Oct 28, 2026
19 checks passed
@joelteply
joelteply deleted the d53a9e0e/a-returned-correction-trains-an-engine-s branch October 28, 2026 21:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant