Repository navigation
fix(benchmark): qualify returned legacy budget interruptions - #8221
loopx-agent wants to merge 2 commits into
Conversation
loopx-agent
left a comment
There was a problem hiding this comment.
Reviewer: model_agent; gpt-6-astra; OpenAI; runtime_reported; ultra.
Approval conclusion (author-owned PR; GitHub blocks formal self-approval)
动机 / Motivation
保留旧版 benchmark worker 的操作者需要在真实预算中断后继续原会话。旧 worker 已写入停止回执,恢复入口却只接受仍在执行或额度失败的旧状态,导致完整现场也进不了原会话验证;现在能识别这次已返回的预算停止,并继续检查同一容器、原始文件和真实 native 中断尾。独立验证已覆盖旧格式证据进入共享接收端、原会话继续下一轮,以及 Stop、改写回执和超出原窗口的活动被拒绝。本 PR 不给额外预算、不恢复评分器、不自动重启,也不把部分文件或一次会话读取当成完整恢复成功。
改动思路 / Approach
沿原有 Docker source proof 和共享 interruption reader 修正旧状态兼容,避免另造恢复调度器。本次只打通已返回预算停止的原会话验证;完整源、评分器机会和新的有限预算仍由原 owner 独立准入。先检查原容器已停止且重启命令惰性,再核原 scheduler、待返回标记、唯一末回执和全部参与文件。对于新接纳的旧状态,原生过程上界取回执秒数,而不是后来维护时停止容器的时间。之后仍由原 native reader 检查真实中断尾,由当前 Goal/session admission 决定是否允许下一轮;没有另造重试或预算 owner。
具体改动 / Changes
关键代码讲解
prepare_stopped_recovery(benchmark/edgebench/stopped_recovery.py)增加确切的历史 stopped/75 分支:不允许现代 attempt ID、未知 failure、成功回执或预派发预算短路。调用者提供的 session 必须与原回执一致;最新回执排序和 quota-failed 路径共用,原字节不重写。
read_interrupted_host(benchmark/runtime/interruption_recovery.py)仅在这种已返回的旧状态中,允许原 hard deadline 晚于回执,兼容旧 worker 预留收尾时间;同时重新核对回执时间,原 native_interrupted_turn 继续拒绝窗口之外的 native 活动。其余 started/modern 路径保留原时间限制。
基准合同是目标分支的 retained-filesystem continuation:unchanged-source、interrupted-native 和 separate-authority 三项均落在原 owner 上。README 明确兼容条件;没有改变普通 completed checkpoint、评分机会或新的有限预算准入。
对主干的风险 / Risks
PASS:独立复跑 69 项 focused tests,包含真实 SQLite、scheduler/worker 子进程和 synthetic CLI;两个 opt-in Docker 用例另在隔离 Linux 中实际执行。独立 base/head 配对显示 started、过期条件及 quota-failed 路径的返回、原字节和权限标记一致;旧 returned75 的接受是明确的预期变化。预派发短路仍拒绝,回执仅修改时间也被接收端拒绝。Stop、改变 profile/session、晚到 native 活动与跨容器复制证明均有拒绝覆盖。
初次本地/guest 测试分别遇到导入路径与非物理临时路径的 fixture 问题,保留后以精确工作树及物理路径重跑;没有修改产品代码或放宽断言。远端部分 browser CI 失败、其他任务仍待完成,原因未归因;不把它们描述为通过或已证实无关。当前差异的 Python/native 不变量有独立覆盖,合并门仍独立。此处的实际 Docker 是 synthetic source,native 会话读取也不等于模型恢复,更不等于评分器权威已经接回。
语义与 CI 对齐
本次复用现有状态和证明字段,只增加持久旧格式兼容;没有通过扩大预算、改写历史、跳过完整参与文件或清空机会来获得通过。完整 workspace、native scorer 历史和当前执行准入仍是实际恢复的硬条件。
我的整体评价 / Verdict
在 b7f28a6096f6c3b76606776b2b8371b752b1eab5 未发现阻塞问题。对长程工作的改进是让已获授权的旧版预算中断可以走原会话验证,并保留失败、历史和原收尾时间,而不是声称模型成绩改善。沿既有 benchmark CLI 的恢复流程兼容旧状态;没有新增页面、配置项或确认轮次。未执行生产实验恢复,也未验证模型策略收益。保留历史兼容有明确持久化依据,共享最新回执排序的重构足够局部;不能据这次审批跳过独立 scorer、完整源和 finite phase 准入。
English verdict: APPROVE at b7f28a6. No blocking finding after independent source, real SQLite/process, Docker and base/head checks. This is not production recovery or merge authorization.
…ed sessions Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com> (cherry picked from commit d442384) Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com> (cherry picked from commit b7f28a6) Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
b7f28a6 to
3e49d88
Compare
|
Hi @loopx-agent, the DCO If the log confirms a missing |
Older benchmark workers can persist a real model interruption as
stopped/75without an attempt ID. The retained-source producer previously accepted onlystartedor a qualified quota failure, so an otherwise verifiable returned budget stop could not reach the existing same-session recovery path.This extends that producer and its shared receiver to accept the exact legacy returned-stop shape. It requires a stopped inert original container, unchanged participants, a unique latest invoked/timed-out budget receipt, original agent session, and current Goal/session admission. The original scheduler state and receipt remain unchanged. The native window ends at the receipt’s recorded second, rather than a later maintenance container stop; the original hard deadline is retained even if the older worker reserved cleanup time. The native reader must still verify an actual interrupted final Turn with no pending effect.
Validation: 102 focused tests pass, including the real SQLite → scheduler → worker subprocess → native-session recovery path with a synthetic CLI; the two positive paths were re-run after adding an explicit unchanged-receipt assertion. Two network-disabled Docker cases pass for inert proof installation, same-container replay and copied-proof rejection. Negative coverage includes Stop, changed binding/profile, live source, ambiguous/later receipts, new attempt IDs, wrong session, success/unknown failures and native activity after the receipt window. Existing modern interruption and explicit quota-continuation paths remain covered. Premerge has zero direct/selected check failures; its
benchmark_sensitivemaintainer-review hold remains.The change stays in benchmark-toolkit’s existing Docker/native-process provider; it adds no configuration, Core state machine, UI, account operation or automatic retry. It neither grants a new budget nor restores evaluator authority. Read-only qualification of a retained native session does not establish complete workspace recovery or a successful model continuation. Full source qualification and independent evaluator history/opportunity/cooldown restoration remain execution prerequisites. No model or live experiment was started by this change.
Target is the existing integration branch because it owns the stopped-source and finite-continuation prerequisites; this does not merge or deploy to main.
Main target
This proposal now targets
main. Its own five-file change qualifies the returned legacy budget receipt. The only open dependency in this branch is #8217 at8bc878b29c8b3eda2c6b0ade4c0ed496bb4476b7. Review the five-file comparison separately. Once #8217 lands, rebase this slice onto main and verify the new exact head before merge. Independent legacy receipt slice