Skip to content

fix(benchmark): qualify returned legacy budget interruptions - #8221

Open
loopx-agent wants to merge 2 commits into
mainfrom
codex/legacy-returned-budget-continuation-20261107
Open

loopx-agent wants to merge 2 commits into
mainfrom
codex/legacy-returned-budget-continuation-20261107

Conversation

@loopx-agent

@loopx-agent loopx-agent commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

Older benchmark workers can persist a real model interruption as stopped/75 without an attempt ID. The retained-source producer previously accepted only started or a qualified quota failure, so an otherwise verifiable returned budget stop could not reach the existing same-session recovery path.

This extends that producer and its shared receiver to accept the exact legacy returned-stop shape. It requires a stopped inert original container, unchanged participants, a unique latest invoked/timed-out budget receipt, original agent session, and current Goal/session admission. The original scheduler state and receipt remain unchanged. The native window ends at the receipt’s recorded second, rather than a later maintenance container stop; the original hard deadline is retained even if the older worker reserved cleanup time. The native reader must still verify an actual interrupted final Turn with no pending effect.

Validation: 102 focused tests pass, including the real SQLite → scheduler → worker subprocess → native-session recovery path with a synthetic CLI; the two positive paths were re-run after adding an explicit unchanged-receipt assertion. Two network-disabled Docker cases pass for inert proof installation, same-container replay and copied-proof rejection. Negative coverage includes Stop, changed binding/profile, live source, ambiguous/later receipts, new attempt IDs, wrong session, success/unknown failures and native activity after the receipt window. Existing modern interruption and explicit quota-continuation paths remain covered. Premerge has zero direct/selected check failures; its benchmark_sensitive maintainer-review hold remains.

The change stays in benchmark-toolkit’s existing Docker/native-process provider; it adds no configuration, Core state machine, UI, account operation or automatic retry. It neither grants a new budget nor restores evaluator authority. Read-only qualification of a retained native session does not establish complete workspace recovery or a successful model continuation. Full source qualification and independent evaluator history/opportunity/cooldown restoration remain execution prerequisites. No model or live experiment was started by this change.

Target is the existing integration branch because it owns the stopped-source and finite-continuation prerequisites; this does not merge or deploy to main.

Main target

This proposal now targets main. Its own five-file change qualifies the returned legacy budget receipt. The only open dependency in this branch is #8217 at 8bc878b29c8b3eda2c6b0ade4c0ed496bb4476b7. Review the five-file comparison separately. Once #8217 lands, rebase this slice onto main and verify the new exact head before merge. Independent legacy receipt slice

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent; gpt-6-astra; OpenAI; runtime_reported; ultra.

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

动机 / Motivation

保留旧版 benchmark worker 的操作者需要在真实预算中断后继续原会话。旧 worker 已写入停止回执,恢复入口却只接受仍在执行或额度失败的旧状态,导致完整现场也进不了原会话验证;现在能识别这次已返回的预算停止,并继续检查同一容器、原始文件和真实 native 中断尾。独立验证已覆盖旧格式证据进入共享接收端、原会话继续下一轮,以及 Stop、改写回执和超出原窗口的活动被拒绝。本 PR 不给额外预算、不恢复评分器、不自动重启,也不把部分文件或一次会话读取当成完整恢复成功。

改动思路 / Approach

沿原有 Docker source proof 和共享 interruption reader 修正旧状态兼容,避免另造恢复调度器。本次只打通已返回预算停止的原会话验证;完整源、评分器机会和新的有限预算仍由原 owner 独立准入。先检查原容器已停止且重启命令惰性,再核原 scheduler、待返回标记、唯一末回执和全部参与文件。对于新接纳的旧状态,原生过程上界取回执秒数,而不是后来维护时停止容器的时间。之后仍由原 native reader 检查真实中断尾,由当前 Goal/session admission 决定是否允许下一轮;没有另造重试或预算 owner。

具体改动 / Changes

关键代码讲解

prepare_stopped_recovery(benchmark/edgebench/stopped_recovery.py)增加确切的历史 stopped/75 分支:不允许现代 attempt ID、未知 failure、成功回执或预派发预算短路。调用者提供的 session 必须与原回执一致;最新回执排序和 quota-failed 路径共用,原字节不重写。

read_interrupted_host(benchmark/runtime/interruption_recovery.py)仅在这种已返回的旧状态中,允许原 hard deadline 晚于回执,兼容旧 worker 预留收尾时间;同时重新核对回执时间,原 native_interrupted_turn 继续拒绝窗口之外的 native 活动。其余 started/modern 路径保留原时间限制。

基准合同是目标分支的 retained-filesystem continuation:unchanged-source、interrupted-native 和 separate-authority 三项均落在原 owner 上。README 明确兼容条件;没有改变普通 completed checkpoint、评分机会或新的有限预算准入。

对主干的风险 / Risks

PASS:独立复跑 69 项 focused tests,包含真实 SQLite、scheduler/worker 子进程和 synthetic CLI;两个 opt-in Docker 用例另在隔离 Linux 中实际执行。独立 base/head 配对显示 started、过期条件及 quota-failed 路径的返回、原字节和权限标记一致;旧 returned75 的接受是明确的预期变化。预派发短路仍拒绝,回执仅修改时间也被接收端拒绝。Stop、改变 profile/session、晚到 native 活动与跨容器复制证明均有拒绝覆盖。

初次本地/guest 测试分别遇到导入路径与非物理临时路径的 fixture 问题,保留后以精确工作树及物理路径重跑;没有修改产品代码或放宽断言。远端部分 browser CI 失败、其他任务仍待完成,原因未归因;不把它们描述为通过或已证实无关。当前差异的 Python/native 不变量有独立覆盖,合并门仍独立。此处的实际 Docker 是 synthetic source,native 会话读取也不等于模型恢复,更不等于评分器权威已经接回。

语义与 CI 对齐

本次复用现有状态和证明字段,只增加持久旧格式兼容;没有通过扩大预算、改写历史、跳过完整参与文件或清空机会来获得通过。完整 workspace、native scorer 历史和当前执行准入仍是实际恢复的硬条件。

我的整体评价 / Verdict

在 b7f28a6096f6c3b76606776b2b8371b752b1eab5 未发现阻塞问题。对长程工作的改进是让已获授权的旧版预算中断可以走原会话验证,并保留失败、历史和原收尾时间,而不是声称模型成绩改善。沿既有 benchmark CLI 的恢复流程兼容旧状态;没有新增页面、配置项或确认轮次。未执行生产实验恢复,也未验证模型策略收益。保留历史兼容有明确持久化依据,共享最新回执排序的重构足够局部;不能据这次审批跳过独立 scorer、完整源和 finite phase 准入。

English verdict: APPROVE at b7f28a6. No blocking finding after independent source, real SQLite/process, Docker and base/head checks. This is not production recovery or merge authorization.

…ed sessions

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
(cherry picked from commit d442384)
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
(cherry picked from commit b7f28a6)
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
@loopx-agent
loopx-agent force-pushed the codex/legacy-returned-budget-continuation-20261107 branch from b7f28a6 to 3e49d88 Compare November 7, 2026 09:33
@loopx-agent
loopx-agent changed the base branch from codex/edgebench-integration-r31-20261107 to main November 7, 2026 09:33
@mergify

mergify Bot commented Nov 7, 2026

Copy link
Copy Markdown

Hi @loopx-agent, the DCO Sign-off check did not pass. Please inspect
its details first: checkout, fetch, timeout or infrastructure errors
need their own recovery, not a rewrite of otherwise signed commits.

If the log confirms a missing Signed-off-by trailer, amend the
affected commit with git commit --amend -s; for multiple commits,
use an interactive rebase against the current base from the correct
base-repository remote and sign off each affected commit. Push the
rewritten PR branch with git push --force-with-lease origin HEAD.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant