Skip to content

knowledge(vllm-omni): 1 change(s) from intake #428

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tzhouam merged 1 commit into from
Oct 26, 2026
Merged

knowledge(vllm-omni): 1 change(s) from intake #428

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion knowledge/repos/vllm-omni/rules-duplexeval-judge.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ created: 2026-10-26
updated: 2026-10-26
type: rule
tags: [vllm-omni]
sources: ["PR #9390"]
sources: ["PR #9390", "PR #9056"]
---

# vllm-omni Omni-DuplexEval 判分校验规则
Expand All @@ -17,3 +17,12 @@ sources: ["PR #9390"]
- 验收：`tests/benchmarks/duplex/test_omni_duplex_eval_judge.py` 以 `core_model and cpu` 标记通过，覆盖无效分数参数化拒绝、合法官方格式保留、不完整回复无分数但轨迹含 raw_response 与被拒 finish_reason、缺失或非 stop 完成状态即使分数有效也被拒、失败重试归档旧分数并从摘要排除且无关样本仍计数；`omni_duplex_eval_eval.py` 保留 content 判分失败后的 `frame-sample` 回退；`docs/cli/bench/serve.md` 保留 finish_reason 与按任务分数字段要求及 `.judge.jsonl`、`.previous` 工件说明。^[PR #9390]

<!-- kb:rule status=active since=v0.30.0 -->

## VLLM-OMNI-PR9056-JUDGE-GT-REFS — Omni-DuplexEval content 提示词的 GT 参考须先去空白、去空串并按首现去重

- 触发：改动 `vllm_omni/benchmarks/duplex/omni_duplex_eval_eval.py` 中 `evaluate_sample` 传给 `build_content_prompt` 的参考集合或 `_gt_references` 辅助函数，或改动 `tests/benchmarks/duplex/test_omni_duplex_eval_gt_references.py` 回归测试；本规则仅覆盖 content 提示词的 GT 参考处理，temporal 与 proactive-reminder 判定路径不在范围内。
- 强制：content 提示词调用点必须传入 `_gt_references([sample.answer1, sample.answer2])` 的结果：跳过 `None`，对每项做 `str(...).strip()`，丢弃剥离后为空的串，再以 `list(dict.fromkeys(...))` 按首次出现去重，对齐官方 OpenBMB/Omni-DuplexEval 评测器 `non_empty_strings` 加首现去重的参考语义；`build_content_prompt` 的 pin 措辞与 `PROTOCOL_PIN` 不随该过滤改动。
- 禁止：把 `[sample.answer1, sample.answer2]` 未经 `_gt_references` 直接入参（空 `answer2` 会在 `Reference annotations` 块留下空编号槽，`answer1 == answer2` 会把同一参考列两次）；删除或削弱该过滤（如只去空串不去重）；借机改动 content 提示词 pin 措辞、`PROTOCOL_PIN` 或 temporal/proactive-reminder 路径。
- 验收：`tests/benchmarks/duplex/test_omni_duplex_eval_gt_references.py`（`core_model` + `cpu` 标记）通过：`answer2 == ""`、`answer1 == answer2`、参考带首尾空白三类参数下，捕获的 content 提示词均只含一个 `Reference annotations` 条目、无 `\n2. ` 编号槽且参考文本仅出现一次。^[PR #9056]

<!-- kb:rule status=active since=v0.30.0 -->
Loading