Skip to content

docs: publish the m2 authorization report on dataset v3 - #30

Merged
poppycoderr merged 1 commit into
mainfrom
docs/m2-report
Oct 2, 2026
Merged

poppycoderr merged 1 commit into
mainfrom
docs/m2-report

Conversation

@poppycoderr

Copy link
Copy Markdown
Owner

Background

Milestone M2 ends with a report on the final dataset of the milestone. This PR commits the main CI run on dataset v3 and updates the READMEs, the results chart and the roadmap to match it.

What is published

 benchmarks/reports/
 ├── m1a-baseline/            # dataset v1
 ├── m1b-hybrid/              # dataset v1
 ├── m2-labelled-dataset/     # dataset v2
+└── m2-authorization/        # dataset v3: the evaluation-results artifact of the main CI run at 8ccf91a, unchanged

test split, 65 answerable cases, Linux x86_64 (AMD EPYC 7763):

Strategy Recall@10 MRR@10 nDCG@10 Violations Scope failures
sparse-only 0.923 [0.85, 0.98] 0.683 [0.59, 0.77] 0.742 [0.66, 0.82] 0 0
dense-only 0.969 [0.92, 1.00] 0.873 [0.80, 0.94] 0.898 [0.84, 0.95] 0 0
hybrid-rrf 0.969 [0.92, 1.00] 0.822 [0.74, 0.89] 0.857 [0.79, 0.91] 0 0
bm25-reference 0.931 [0.87, 0.98] 0.720 [0.63, 0.80] 0.770 [0.69, 0.84] 0 0

What the report supports

  • Authorization. There are zero unauthorized results across 122 cases, 31 of which are authorization negatives. The full chunk listings of all five principals match the hand-written visibility.
  • Scope. There are zero scope failures across 13 scope cases. Scope failures are counted apart from security violations.
  • Ranking.
    • Dense beats full-text search on MRR@10: +0.19 [+0.11, +0.28].
    • Hybrid shows no detectable difference from dense: −0.05 [−0.11, +0.01].
    • Full-text search shows no detectable difference from BM25: +0.04 [−0.02, +0.10]. The claim made on dataset v1 stays withdrawn.

Also in this PR

  • READMEs: the results table gains a scope column, the numbers move to v3, and the roadmap marks M2 as done.
  • Diagrams: the results chart and the roadmap are regenerated from scripts/generate-diagrams.py with the new numbers.
  • Not included: a tag for M2. That is a release and waits for the maintainer.

背景

里程碑 M2 以一份报告收尾,报告用的是这个里程碑最终的数据集。本 PR 提交 main 上 CI 在数据集 v3 上的运行结果,并让 README、结果图和路线图与它保持一致。

发布了什么

目录变化和指标见英文部分。m2-authorization/ 是 main 在 8ccf91a 上那次 CI 运行的 evaluation-results 产物,原样提交。

这份报告能支持的结论

  • 授权。 122 条用例的越权结果为 0,其中 31 条是授权负例。5 个身份能列出的全部 chunk,都与手写的可见性标注一致。
  • 适用范围。 13 条范围用例的范围失败为 0。范围失败与安全越权分开统计。
  • 排序。
    • dense 在 MRR@10 上优于全文检索:+0.19 [+0.11, +0.28]。
    • hybrid 与 dense 之间没有可检测的差异:−0.05 [−0.11, +0.01]。
    • 全文检索与 BM25 之间没有可检测的差异:+0.04 [−0.02, +0.10]。在数据集 v1 上得出的那条结论维持撤回。

本 PR 的其他改动

  • README: 结果表增加了「范围失败」一列,数字换成 v3,路线图把 M2 标为已完成。
  • 配图: 结果图和路线图由 scripts/generate-diagrams.py 用新数字重新生成。
  • 不包含: M2 的标签。那属于发布操作,等维护者决定。

@poppycoderr
poppycoderr merged commit dc54a83 into main Oct 2, 2026
6 checks passed
@poppycoderr
poppycoderr deleted the docs/m2-report branch October 2, 2026 17:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant