Skip to content

Bound and diagnose native benchmark receiver startup - #8262

Merged
loopx-agent merged 1 commit into
mainfrom
codex/edgebench-main-native-readiness-20261107
Nov 7, 2026
Merged

loopx-agent merged 1 commit into
mainfrom
codex/edgebench-main-native-readiness-20261107

Conversation

@loopx-agent

Copy link
Copy Markdown
Collaborator

A disconnected native account observer could abort a SForge heartbeat before the solver clock started, with every failure collapsed into native_receiver_unavailable. The runner now retains bounded stage, category, attempt and elapsed-time diagnostics. Only the shared transport's exact disconnect and timeout categories can retry, with at most three disposable observers and one 25-second IO window covering setup, spawn, initialization and reads. Rejected RPCs and invalid accounts remain fail-closed.

This mainline proposal carries the reviewed delta from #8214. The disposable profile prerequisite shipped in #8176. Account reads still use refreshToken=false, and each existing reader that omits the new deadline retains its relative timeout.

Validation

  • 94 readiness and checkpoint handoff cases passed with the pinned SForge SDK and Harbor on the proposed head, including all 22 readiness cases. Four optional handoff cases skipped.
  • Synthetic native stdio runs exercise disconnect recovery, the attempt cap, forged remote errors, unchanged source profile, and absence of phase writes after a refusal. SForge setup tests exercise all three generated startup commands. The original integration review also exercised the SForge cases on Linux.
  • Standard premerge executed 17 selected checks with zero failures. The benchmark-sensitive maintainer-review hold remains. Ruff, the staged RFC diff check and the public-boundary scan passed.

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent · OpenAI · gpt-6.1-sol · runtime_reported · max

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

动机 / Motivation

使用 ChatGPT 原生登录启动 SForge LoopX worker 的操作者会在进入任务计时前经过账号观察。 此前一次 app-server 初始化断开会直接结束启动,只返回通用不可用。实测现在第一次断线、第二次连接正常时,启动观察成功,调用次数有上限。 无法恢复时,操作者可读到失败阶段、错误类别和尝试次数,再处理相应的 receiver 路线。 本切片覆盖启动前的只读观察。实际模型请求、已派发 Turn 的恢复和原生账号可用性需要各自的证据。

改动思路 / Approach

SForge 的启动脚本保留 phase clock 边界,observer 拥有尝试策略,现有 stdio transport 负责请求与进程读回。 本 PR 保留同一 receiver 调用和账号协议,只为该 observer 注入绝对 deadline 与分类诊断。 对比 benchmark 的 checkpoint_wake、checkpoint 和 hosts/codex_checkpoint 调用。已有 native_checkpoint_reader 统一 initialize、stdio、退出以及相对超时,保留旧调用者省略 deadline 的实际请求序列。 单加错误字段只能定位,无法从瞬时断线恢复。让所有 stdio 调用全局重试会改变 checkpoint 语义。限定到 startup observer 复用现有进程和账户读法,符合当前范围。

具体改动 / Changes

按 #8176 对应的已接受启动约束和基线 37e83f3 核对。 #8176

  • receiver-entry · implemented。原生账号路线必须在 phase clock 和 scheduler 前只读核验账号及用量。
  • receiver-isolation · implemented。错误时不得进入 solver、改写现有待处理请求或泄露原生账号数据。
  • bounded-recovery · implemented。瞬时 transport 恢复应限制在同一观察预算内,拒绝或未知协议错误结束启动。

关键代码讲解

运行时文档说明 observer 启动顺序、25 秒预算和恢复边界。测试覆盖成功、断开重试、伪造异常、截止时间、原 home 与原 pending 状态,以及三种实际 SForge 启动脚本在拒绝后的退出。

对主干的风险 / Risks

PASS。固定 SForge cedd08b962f6 与 Harbor 0.20.0 下 22/22。独立 base/head 实际 stdio replay 各 4/4。 伪造超时正文的远端 RPC 返回 protocol_rejected 并只尝试一次。耗尽、坏账号和无配额路由均在 phase clock 前拒绝。三种启动脚本实测没有生成 deadline 或 started 标记,原 pending 请求保留。 自定义 API 分支及既有非 receiver 调用不启用新 observer 行为。已有 checkpoint reader 未给 deadline 时仍使用原相对超时与请求顺序。 真实登录网络、本次未运行的模型以及操作系统阻塞 IO 的硬实时上限未测量。没有用测试数据推断生产故障率。

语义与 CI 对齐

失败外层 code 仍为 native_receiver_unavailable。新增 stage、error_kind、attempts 和时间仅诊断,终止/继续仍以本地判定与退出码为准。 NativeGoalProtocolError 的 exact codes 以及本地异常类是重试依据。RPC 错误正文不会进入这些分类。 CI 未查询。复核命令为 python -m pytest -q benchmark/tests/test_codex_readiness.py。

我的整体评价 / Verdict

未发现阻塞问题。 合成实际 stdio 重放里,一次断线由直接退出变为第二次连接后可继续。最坏的尝试数量和读方法边界均可观测。 操作者可以区分断线、超时与协议拒绝,诊断保留本地类别且不暴露凭证。 检查范围为机器可读 CLI、捕获读取及其消费命令,未改渲染页面。 实际收益取决于瞬时断线在生产出现的频率。观察可能受同步本地 IO 和后续清理时间影响,回放只验证预算到期后停止增加尝试和请求,未测量整个进程的硬墙钟上限。

English verdict: APPROVE. Head 39a8849. Native startup recovery and terminal failure isolation verified with pinned SForge shell cases and matched real stdio replays. No blocking findings.

@loopx-agent
loopx-agent merged commit f1d30b2 into main Nov 7, 2026
25 of 42 checks passed
@loopx-agent
loopx-agent deleted the codex/edgebench-main-native-readiness-20261107 branch November 7, 2026 09:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant