feat(data): add access labels and authorization cases as dataset v2, with a stricter security gate

README.md

@@ -46,7 +46,7 @@ The outsider gets no "permission denied", no hit count and no Northstar document
 | Authorization in the retrieval query | Tenant, clearance, department and project rules compiled once per request into the SQL of every channel; property-tested against a reference evaluator | Scope filters, audit events, labelled evaluation data (M2) |
 | Retrieval | `sparse-only` (PostgreSQL FTS), `dense-only` (exact pgvector) and `hybrid-rrf` (reciprocal rank fusion with overlap deduplication); every response carries a plan hash | Cross-encoder reranking (M3) |
 | Ingestion | Asynchronous jobs (`202` + poll) with a `SKIP LOCKED` worker, bounded retry and resume; content-hash versioning; Markdown and plain-text chunking with sentence-level splitting and overlap; disable and delete apply to the next query, with background cleanup | – |
-| Evaluation | 70 cases over 21 documents, hard negatives, a BM25 reference row, bootstrap intervals and paired comparisons, security gate in CI | Label-level authorization negatives (M2) |
+| Evaluation | 108 cases over 28 labelled documents: paraphrases, hard negatives and 30 authorization negatives; a BM25 reference row, bootstrap intervals and paired comparisons; a security gate in CI that checks every returned chunk and every principal's full listing | Scope and time-dependent cases (M2) |
 | Answers | Ranked evidence from `/api/v1/retrieval/search` | `/api/v1/query` with citations and abstention (M3) |
 | Operations | Docker Compose, CI on every PR | OpenTelemetry traces, audit events, dashboards (M2–M4) |
 
@@ -85,13 +85,13 @@ Most RAG demos retrieve first and filter afterwards. That leaks rows into applic
     <img src="./assets/diagrams/ga-authorization.en.svg" alt="Authorization is a retrieval concern, not a post-filter" />
 </p>
 
-The predicate carries the tenant, clearance, department and project rules. The demo corpus has no access labels yet, so this demo shows the tenant rule. The shape of the solution is the point: whatever the rules are, they belong in the query that selects candidates.
+The predicate carries the tenant, clearance, department and project rules; this demo shows the tenant rule. The shape of the solution is the point: whatever the rules are, they belong in the query that selects candidates.
 
 ## Retrieval evaluation
 
 [`benchmarks/reports/m1b-hybrid/`](./benchmarks/reports/m1b-hybrid/) holds the committed run: `run.json` (dataset version, commit, strategies, policy version, bootstrap seed, platform), `cases.jsonl` (per-case rankings) and the rendered `report.md`. Regenerate it with `./scripts/benchmark --out benchmarks/reports/<name>`.
 
-`test` split, 47 answerable cases, 95% bootstrap intervals, produced by the CI runner (Linux x86_64):
+Dataset v1 (21 unlabelled documents, 70 cases; the labelled dataset v2 is newer than this report), `test` split, 47 answerable cases, 95% bootstrap intervals, produced by the CI runner (Linux x86_64):
 
 | Strategy | Recall@10 | MRR@10 | nDCG@10 | Security violations |
 |---|---|---|---|---|
@@ -157,7 +157,7 @@ Guards in place today:
 - an architecture test — only `AuthorizedChunkQuery` may read the chunk table;
 - a property-based test — random principals and labels, with every query path compared against a separate reference evaluator;
 - integration tests on real pgvector — each rule of the decision table on all three strategies, cross-tenant isolation, hostile claim values, invalid and under-scoped tokens;
-- the evaluation security gate — returned documents are compared against hand-labelled visibility, never against the compiler itself.
+- the evaluation security gate — every returned chunk is checked for document and version against hand-labelled visibility, never against the compiler itself, and each principal's full chunk listing must match its visible set.
 
 Planned with M2: labelled evaluation data, a gate that checks version and chunk granularity, and audit events.
 

README.zh-CN.md

@@ -46,7 +46,7 @@ mallory-outsider (tenant external) · dense-only · policy abac/1
 | 检索查询内的授权 | 租户、密级、部门、项目四条规则，每次请求编译一次，写进每条通道的 SQL；用基于属性的测试对照参考实现验证 | 适用范围过滤、审计事件、带标签的评测数据（M2） |
 | 检索 | `sparse-only`（PostgreSQL FTS）、`dense-only`（pgvector 精确检索）与 `hybrid-rrf`（RRF 融合并去除重叠 chunk）；每个响应带检索配置哈希 | cross-encoder 重排（M3） |
 | 入库 | 异步任务（`202` + 轮询），`SKIP LOCKED` worker、有限重试与断点续跑；内容哈希版本管理；Markdown 与纯文本切分，按句拆分长段落并带重叠；停用与删除对下一次查询生效，后台清理 | – |
-| 评测 | 21 篇文档 70 条用例、hard negatives、BM25 参考行、bootstrap 置信区间与配对比较、CI 安全门禁 | 标签级授权负例（M2） |
+| 评测 | 28 篇带标签的文档、108 条用例：改写、hard negatives 与 30 条授权负例；BM25 参考行、bootstrap 置信区间与配对比较；CI 安全门禁逐个检查返回的 chunk 和每个身份的完整可见列表 | 适用范围与时间相关的用例（M2） |
 | 回答 | `/api/v1/retrieval/search` 返回排序后的证据 | 带引用与拒答的 `/api/v1/query`（M3） |
 | 运维 | Docker Compose、每个 PR 的 CI | OpenTelemetry trace、审计事件、dashboard（M2–M4） |
 
@@ -85,13 +85,13 @@ curl -s localhost:8080/api/v1/retrieval/search -H "Authorization: Bearer $TOKEN"
     <img src="./assets/diagrams/ga-authorization.zh-CN.svg" alt="授权属于检索环节，不是事后过滤" />
 </p>
 
-谓词里已经包含租户、密级、部门和项目四条规则。demo 语料还没有访问标签，所以这个演示展示的是租户规则。重点是解法的形状：无论规则是什么，它都应该待在筛选候选的那条查询里。
+谓词里已经包含租户、密级、部门和项目四条规则；这个演示展示的是租户规则。重点是解法的形状：无论规则是什么，它都应该待在筛选候选的那条查询里。
 
 ## 检索评测
 
 [`benchmarks/reports/m1b-hybrid/`](./benchmarks/reports/m1b-hybrid/) 保存了提交在仓库中的运行结果：`run.json`（数据集版本、commit、策略、policy 版本、bootstrap 种子、运行平台）、`cases.jsonl`（逐用例排名）与渲染出的 `report.md`。用 `./scripts/benchmark --out benchmarks/reports/<名称>` 可重新生成。
 
-`test` 划分，47 条可回答用例，95% bootstrap 区间，由 CI runner（Linux x86_64）生成：
+数据集 v1（21 篇无标签文档、70 条用例；带标签的数据集 v2 比这份报告新），`test` 划分，47 条可回答用例，95% bootstrap 区间，由 CI runner（Linux x86_64）生成：
 
 | 策略 | Recall@10 | MRR@10 | nDCG@10 | 越权结果 |
 |---|---|---|---|---|
@@ -157,7 +157,7 @@ AND (cardinality(v.required_projects)  = 0 OR v.required_projects && CAST(:auth_
 - 架构测试——只有 `AuthorizedChunkQuery` 可以读取 chunk 表；
 - 基于属性的测试——随机生成身份和标签，把每条查询路径的结果与一份独立的参考实现对照；
 - 真实 pgvector 上的集成测试——决策表的每条规则在三种策略上的表现、跨租户隔离、恶意的 claim 值、无效与权限不足的 token；
-- 评测安全门禁——返回的文档与人工标注的可见性比较，而不是与编译器自己比较。
+- 评测安全门禁——每个返回的 chunk 都按文档和版本与人工标注的可见集合比较，绝不与编译器自己比较；每个身份能列出的全部 chunk 也必须与它的可见集合一致。
 
 M2 计划补上：带标签的评测数据、细到版本与 chunk 粒度的门禁、审计事件。
 

benchmarks/README.md

@@ -24,6 +24,8 @@ Keyword (`sparse-only`) and `bm25-reference` rankings are bit-identical on every
 
 `m1b-hybrid` adds the `hybrid-rrf` strategy to the same cases; [docs/evaluation/m1b-hybrid-analysis.md](../docs/evaluation/m1b-hybrid-analysis.md) reads its result. Its single-channel and BM25 rankings are identical to `m1a-baseline`.
 
+`m0-walking-skeleton`, `m1a-baseline` and `m1b-hybrid` use dataset v1. Later reports use dataset v2, which adds labelled documents and authorization cases; numbers across the two versions are not comparable (see the [dataset card](../data/eval/DATASET_CARD.md)).
+
 Runs on the fictional demo corpus are labelled demo benchmarks. They show that the method is reproducible; they say nothing about production retrieval quality.
 
 `results/` at the repository root is the scratch directory for ad-hoc runs and is not tracked.

data/README.md

@@ -5,16 +5,18 @@ Everything in this directory is fictional. Northstar Cloud and Orbit Labs do not
 | Path | Contents |
 |---|---|
 | `corpus/<tenant>/` | The documents themselves, in Markdown |
-| `manifests/<tenant>.yaml` | Document key, title and file for ingestion |
+| `manifests/<tenant>.yaml` | Document key, title, file, access labels and version history for ingestion |
 | `principals.yaml` | Demo identities and the attributes that become token claims |
-| `eval/v1/cases.jsonl` | Evaluation cases: query, principal, evidence quotes, expectations, tags |
-| `eval/v1/visibility.yaml` | Hand-labelled visible documents per principal, used by the security gate |
+| `eval/v2/cases.jsonl` | Evaluation cases: query, principal, evidence quotes, expectations, tags |
+| `eval/v2/visibility.yaml` | Hand-labelled visible documents per principal, used by the security gate |
 | `demo-keys/` | The public demo signing key pair — insecure by design, local demos only |
 
 The corpus and the evaluation labels are published under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). The code in this repository is Apache-2.0.
 
 Orbit Labs deliberately shares vocabulary with Northstar Cloud ("volunteer days", "billing database") so that cross-tenant isolation is tested with wording that would otherwise match.
 
+Load the corpus into a freshly started stack. A document with a version history is ingested oldest version first, so loading it again into the same database adds further versions, and the evaluation then stops with a version mismatch.
+
 Validate the dataset after any change:
 
 ```bash

data/corpus/northstar/eng-atlas-migration-plan.md

@@ -0,0 +1,15 @@
+# Project Atlas: Billing Database Migration Plan
+
+This fictional plan is for members of project Atlas. Atlas moves the billing database to a new storage engine.
+
+## Timeline
+
+The cutover is planned for the second weekend of November. A rehearsal on the staging cluster takes place three weeks before the cutover.
+
+## Rollback
+
+If error rates stay above 2 percent for 10 minutes after the cutover, the Atlas lead switches traffic back to the old primary. The old primary is kept for 14 days before it is decommissioned.
+
+## Freeze
+
+All schema changes to the billing database are frozen from five days before the cutover until the old primary is decommissioned.

data/corpus/northstar/eng-borealis-launch-runbook.md

@@ -0,0 +1,15 @@
+# Project Borealis: Launch Runbook
+
+This fictional runbook is for members of project Borealis, the usage-based pricing engine that has not been announced yet. It is confidential.
+
+## Launch gates
+
+Borealis launches only after the metering pipeline has processed 30 days of shadow traffic with fewer than 5 mismatched invoices per million.
+
+## Kill switch
+
+The feature flag `borealis.metering.enabled` turns the engine off within one minute. Only the Borealis on-call engineer may flip it during the first two weeks after launch.
+
+## Failover
+
+If the metering database fails, usage events are buffered in the queue for up to 6 hours. The engineer does not fail over the billing database for a Borealis outage.

data/corpus/northstar/eng-security-hardening.md

@@ -0,0 +1,15 @@
+# Production Hardening Standard
+
+This fictional standard is for the Northstar Cloud engineering department.
+
+## SSH access
+
+Engineers reach production hosts through the bastion only, and sessions are recorded. Direct SSH from a laptop to a production host is blocked at the network level.
+
+## Secrets
+
+Secrets are stored in the vault and rotated every 90 days. A secret that appears in a log or a chat message is rotated within one hour.
+
+## Patching
+
+Critical security patches are applied to production within 72 hours of release. Other patches follow the monthly maintenance window.

data/corpus/northstar/hr-compensation-bands.md

@@ -0,0 +1,15 @@
+# Compensation Bands
+
+This fictional document lists salary bands for Northstar Cloud engineering roles. It is confidential.
+
+## Salary bands
+
+A senior engineer in Germany is paid between 92,000 and 118,000 EUR per year. A staff engineer in Germany is paid between 121,000 and 149,000 EUR per year.
+
+## On-call pay
+
+Engineers at staff level and above receive an on-call allowance of 400 EUR per week of primary on-call instead of the standard allowance.
+
+## Annual review
+
+Salary bands are reviewed every January. Managers receive the new bands two weeks before they tell their teams.

data/corpus/northstar/sales-pricing-guide.md

@@ -0,0 +1,15 @@
+# Pricing Guide
+
+This fictional guide lists Northstar Cloud list prices and discount limits. From this version on it is for the sales and support departments only.
+
+## List prices
+
+The Standard plan costs 450 EUR per month. The Enterprise plan costs 2,100 EUR per month.
+
+## Discounts
+
+Annual prepayment earns a 10 percent discount on either plan. An account executive may offer up to 15 percent without approval; anything above that needs the sales director.
+
+## Price changes
+
+Existing customers keep their old price until their next renewal.

data/corpus/northstar/sales-pricing-guide.v1.md

@@ -0,0 +1,11 @@
+# Pricing Guide
+
+This fictional guide lists Northstar Cloud list prices.
+
+## List prices
+
+The Standard plan costs 400 EUR per month. The Enterprise plan costs 1,900 EUR per month.
+
+## Discounts
+
+Annual prepayment earns a 10 percent discount on either plan.

data/corpus/northstar/security-key-ceremony.md

@@ -0,0 +1,15 @@
+# Root Key Ceremony
+
+This fictional procedure is restricted to named key custodians.
+
+## Custodians
+
+The root encryption key is split into five shares. Any three custodians together can reconstruct it.
+
+## Storage
+
+Each share is kept in a tamper-evident envelope in a different bank deposit box. The locations are never written down in the same place.
+
+## Schedule
+
+The ceremony is repeated every two years, or immediately after a custodian leaves the company.

data/corpus/northstar/support-escalation-contacts.md

@@ -0,0 +1,15 @@
+# Support Escalation Contacts
+
+This fictional page is for the Northstar Cloud support department only.
+
+## Enterprise escalations
+
+For an Enterprise customer who cannot log in for more than 30 minutes, the agent calls the duty manager on the escalation line instead of opening a ticket.
+
+## Goodwill credits
+
+A support agent may grant a goodwill credit of up to 200 USD without approval. A team lead may grant up to 1,000 USD.
+
+## Abusive contacts
+
+An agent may end a call after one warning if the caller is abusive, and records the reason in the ticket.