feat(retrieval): add region and validity scope filters with asof, compiled apart from authorization

[poppycoderr]

Background

Milestone item 2.3. Region and validity are scope: they decide which documents apply to a query, not who may read them. Open question Q11 blocked this item, because it was undecided what asOf selects. This PR records that decision as ADR-0005 and implements the scope filters.

ADR-0005: what asOf means

asOf filters the current version of each document by its validity window. It never selects an older version, and authorization always uses the labels of the current version. A question about the past is answered by publishing the superseded policy as its own document with a closed validity window.

Selecting historical versions is deferred. It would need rules for which version wins, for labels as they were at the time, for retention of replaced versions, and an evaluation oracle. A mistake in any of them would return content under the wrong labels, which is a leak.

Scope in the query

 where (
   c.tenant_id = :auth_tenant_id and v.classification_rank <= :auth_clearance_rank and …        -- authorization, unchanged
 )
+and (
+  (v.valid_from is null or v.valid_from <= :scope_as_of) and (v.valid_to is null or :scope_as_of < v.valid_to)
+  and (cardinality(v.applies_to_regions) = 0 or :scope_region is null or :scope_region = any(v.applies_to_regions))
+)

• Two conditions, two objects. Scope is compiled by its own type with its own parameters. The authorization predicate is not touched, so the two cannot be confused in code or in metrics.
• Defaults. asOf defaults to the time of the request, and the region defaults to the principal's region claim. A document without regions applies everywhere. A principal without a region gets no region filter.
• The request may set both. They are not authorization: asking about another region or another date changes what is relevant, never what the principal may read.

Contract

POST /api/v1/ingestion-jobs      documents[].appliesToRegions, validFrom, validTo      (optional)
POST /api/v1/retrieval/search    asOf, region                                           (optional); the response echoes "scope"
GET  /api/v1/retrieval/chunks    asOf, region, includeOutOfScope                        (debug scope)

• includeOutOfScope drops the scope condition and never the predicate. The evaluation needs it to compare everything a principal is authorized for with its hand-labelled visible set.
• A scope-only change is handled like a label change: a new version takes over the existing chunks without embedding again, and it applies to the next query.
• Migration V6 adds the three columns to document_version and to queued job documents.

How it is verified

• Property test. It now generates regions, validity windows and scopes as well, including the instants on both boundaries of every window. Every query path must return exactly what a separate reference evaluator allows. Without a scope, it must return exactly the authorized set.
• Integration tests. They cover region defaults and overrides, the validity window, asOf not bringing back a replaced version, and scope never widening authorization.
• Mutations. Making the end of the window inclusive, or dropping the region rule, fails both the property test and the integration tests.

Evaluation impact

None on the metrics, as expected: the demo corpus has no regions or validity windows yet. The CI evaluation run of this PR has zero security violations and the same metric values as the published v2 report.

Of the 432 ranked lists, 430 are identical to the report. The other two are dense-only lists in which two adjacent candidates with near-equal scores swapped places at ranks 7 to 10, with no effect on any metric. All sparse, hybrid and BM25 lists are identical. The scope condition cannot reorder ties, so this is the known sensitivity of near-tied dense scores to the CPU the embeddings were computed on. It had been documented between arm64 and x86_64, and it now appears between two hosted x86_64 runners as well. A follow-up records the CPU model in run.json and updates the reproducibility note.

Scope cases (as_of, region) and separate reporting of scope failures arrive with the next dataset version, together with the documents that exercise them.

---

背景

里程碑的 2.3 项。region 和有效期属于适用范围：它们决定哪些文档适用于一次查询，而不决定谁可以读。这一项此前被开放问题 Q11 卡住，因为 asOf 到底选中什么还没有定论。本 PR 把这个决定记录为 ADR-0005，并实现了范围过滤。

ADR-0005：asOf 的含义

asOf 按有效期过滤每篇文档的当前版本。它不会选出更早的版本，授权始终使用当前版本的标签。要回答关于过去的问题，做法是把被取代的制度作为一篇独立的文档发布，并给它一个封闭的有效期。

选取历史版本这件事推后处理。它需要一系列规则：多个版本并存时取哪个、按当时的标签还是现在的标签判断权限、旧版本保留多久，以及一个评测用的参考答案。其中任何一条出错，都会让内容在错误的标签下被返回，那就是泄漏。

查询里的范围条件

SQL 的变化见英文部分的 diff。

• 两个条件，两个对象。 范围条件由独立的类型编译，使用自己的参数。授权谓词没有被改动，所以两者在代码里和指标里都不会混淆。
• 默认值。 asOf 默认是请求发生的时间，region 默认取身份 token 里的 region。没有标注 region 的文档适用于所有地区。没有 region 的身份不做地区过滤。
• 请求可以指定这两个值。 它们不属于授权：换一个地区或日期来提问，改变的是哪些内容相关，而不是这个身份能读什么。

契约

接口的变化见英文部分的代码块。

• includeOutOfScope 只去掉范围条件，不会去掉授权谓词。评测需要它，才能把一个身份被授权的全部内容和人工标注的可见集合做比较。
• 只改范围 的处理方式和只改标签一样：新版本接管原有的 chunk，不重新计算向量，并对下一次查询生效。
• 迁移脚本 V6 给 document_version 和排队中的任务文档加上了这三列。

如何验证

• 基于属性的测试。 现在还会随机生成 region、有效期和查询范围，包括每个有效期两端边界上的时间点。每条查询路径返回的内容必须和一份独立的参考实现完全一致。不带范围时，返回的必须恰好是被授权的集合。
• 集成测试。 覆盖 region 的默认值与覆盖、有效期、asOf 不会带回已被替换的版本，以及范围条件不会放宽授权。
• 变异检查。 把有效期的结束边界改成闭区间，或者去掉 region 规则，基于属性的测试和集成测试都会失败。

对评测的影响

指标没有变化，这符合预期：demo 语料还没有 region 和有效期。本 PR 的 CI 评测越权结果为 0，各项指标与已发布的 v2 报告相同。

432 个排序列表里有 430 个与报告完全一致。另外 2 个是 dense-only 的列表：两个得分几乎相同的相邻候选在第 7 到第 10 名之间互换了位置，对任何指标都没有影响。sparse、hybrid 和 BM25 的列表全部一致。范围条件不可能改变并列候选的顺序，所以这是已知的现象：得分接近的 dense 结果对计算向量的 CPU 敏感。之前记录的是 arm64 和 x86_64 之间的差异，现在在两台托管的 x86_64 runner 之间也出现了。后续会在 run.json 里记录 CPU 型号，并更新可复现性的说明。

带 as_of 和 region 的范围用例，以及范围失败的单独统计，会随下一个数据集版本和对应的文档一起加入。