Repository navigation
A returned correction trains: an engine still loading refuses before any job exists (dispatch retries), and a pass's return dispatches its bucket below threshold (card d53a9e0e, gate 3) - #5575
Conversation
…any job exists (dispatch retries), and a pass's return dispatches its bucket below threshold (card d53a9e0e, gate 3) The windows 5090 after b638145 (Codex, 2026-10-28 20:16-20:22Z): #5563 returned Kimi's canonical jobs, then (1) re-dispatched job c9e5b989 failed in 2 ms on /train/measure 'HTTP 503 Loading model', ended FAILED with no partition and stranded its examples; (2) her code/owner bucket held 11 returned examples under minExamples 16, never firing. (1) measure_examples types its failure (MeasureError::NotReady for HTTP 503 or a refused connection). create_job checks the served-window measure before request.json, a run or a partition exist and refuses TrainerUnavailable, the trigger's retryable refusal: the batch stays owned and recovery retries it once the engine serves. (2) A recovery pass that returns examples (an ended job's parked remainder, an engine-refused job) dispatches that bucket at once through the flush verb's own dispatch (dispatch_returned): they met a threshold when first dispatched. A return happens only when her window can train one or a smaller chunk is left, so no loop. Tests: fake_lane 'loading' (503): TrainerUnavailable, nothing posted, no job dir; a returned batch below threshold dispatches once, whole, in order. Local (enrico, -j2): continuum-dreaming training_trigger 83/0, continuum-genome engine_lora_adapter 47/0; check --lib clean in both crates. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE
|
Review at 5f9bd4e: blocking recovery gap. [P1] Make returned-batch dispatch eligibility durable. return_parked_remainder commits acceptance/return before remainders.rs calls dispatch_returned. A crash or cancellation between them, or the run_owned error branch, leaves the below-threshold batch accepted without an active dispatch. On restart, AlreadyReturned is ignored; recover_pass calls dispatch_pending only when ready_to_dispatch is true, which still requires the original threshold without an active dispatch. The same stranded-return failure therefore survives this patch across restart. Persist an idempotent returned-dispatch intent or eligibility with acceptance, and consume it through normal recovery. Avoid indiscriminately flushing historical returns. Add a fixture that restarts after return acceptance but before dispatch, then runs recovery twice and proves one complete batch is dispatched exactly once. The current test calls the helper twice in the same state and misses this boundary. The early typed TrainerUnavailable handling for HTTP503/connect failure looks appropriate in source review. Owner reports 83 dreaming and47 adapter tests passing; I did not execute those locally. No adoption/learning verdict yet. |
…arries 'returned', and recovery dispatches its bucket below threshold, once across restarts (Codex on #5575) The flush after a return ran after its receipt; a crash between the two left the examples below threshold with AlreadyReturned, the same strand. Now PendingBatch.returned is persisted with the submission that accepts a return (submit_batch_inner, returned_job), kept exact per part through split (submission_returned), and ready_to_recover (the recovery tick only) dispatches a bucket holding one. Submit keeps its threshold; rows from before this read false, so no historical bucket is flushed. dispatch_returned is deleted: one dispatch path. Fixture: return accepted, core dies before any pass; owner 2's recovery dispatches it whole, owner 3's dispatches nothing more; a plain submit below threshold stays held. Mutation (ready_to_recover -> ready_to_dispatch) fails it at owner 0. Local (enrico, -j2): continuum-dreaming training_trigger 83/0; check --lib clean. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE
|
Source review PASS at 24c01a7; the earlier return-to-dispatch hold is resolved. Eligibility now persists in the accepted submission, survives hydration, and is consumed by normal recovery without bypassing an in-flight job/trial. Per-submission flags preserve returned eligibility across budget splits. Historical submissions default false. The replacement fixture restarts after acceptance but before dispatch, recovers through two owners, verifies the whole return is sent once, and checks that an ordinary below-threshold submission stays held. The early 503/connect TrainerUnavailable path still creates no job artifacts, retaining retry ownership in the dispatch. Owner reports 83 trigger and47 adapter tests passing; displayed Windows check and hygiene pass, remaining CI is in progress. This is source approval, conditional on required checks. No live adoption or learning-gate pass claimed. |
The windows 5090 after b638145 (Codex, 2026-10-28 20:16-20:22Z): #5563 returned Kimi's
canonical jobs, then (1) re-dispatched job c9e5b989 failed in 2 ms on /train/measure
'HTTP 503 Loading model', ended FAILED with no partition and stranded its examples;
(2) her code/owner bucket held 11 returned examples under minExamples 16, never firing.
(1) measure_examples types its failure (MeasureError::NotReady for HTTP 503 or a
refused connection). create_job checks the served-window measure before request.json,
a run or a partition exist and refuses TrainerUnavailable, the trigger's retryable
refusal: the batch stays owned and recovery retries it once the engine serves.
(2) A recovery pass that returns examples (an ended job's parked remainder, an
engine-refused job) dispatches that bucket at once through the flush verb's own
dispatch (dispatch_returned): they met a threshold when first dispatched. A return
happens only when her window can train one or a smaller chunk is left, so no loop.
Tests: fake_lane 'loading' (503): TrainerUnavailable, nothing posted, no job dir;
a returned batch below threshold dispatches once, whole, in order.
Local (enrico, -j2): continuum-dreaming training_trigger 83/0, continuum-genome
engine_lora_adapter 47/0; check --lib clean in both crates.
Co-Authored-By: Claude Opus 5.5 (1M context) noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE