feat(sleep): add adversarial candidate probes

[Bogdan (Dan) Baciu (bogdanbaciu21)]

What Problem This Solves

A candidate skill can improve its held-out validation score while depending on the exact wording or framing of harvested requests. It can therefore pass the ordinary gate and still break when the same request arrives with a harmless surface change.

The first revision of these probes scored only the candidate, so blocking mode could not distinguish brittleness the candidate introduced from prompt-frame sensitivity already present in the backend, and a single stochastic sample could mark a row brittle at the default margin.

Why This Change Was Made

This revision makes the decision baseline-relative and stochastically defensible before any probe result can block adoption.

Identical source and probe pairs are scored under both the current documents and the candidate documents, each score is the mean of a configured number of repeated rollouts, and a row is brittle only when the candidate's probe minus source gap worsens beyond the margin relative to the baseline gap and the worsening holds in a strict majority of rollout indices. Blocking additionally requires at least two rollouts, so one stochastic sample can never reject a candidate; advisory runs may use one. The baseline documents are required arguments, so no caller can silently compare against an unintended baseline. Request-frame transformations are restricted to a defensible semantic-preservation contract: only explicitly politeness-marked requests are reframed, and ability, permission, and desire questions are never touched.

Project Fit

• Improves SkillOpt-Sleep by making candidate brittleness visible and attributable before adoption.
• Supports stable gains across varied request phrasing while preserving existing behavior when the feature is disabled.

User Impact

Operators can see which harmless request variation broke a staged candidate, and whether the breakage is candidate-introduced or pre-existing backend sensitivity. Advisory mode adds evidence without changing gate decisions; explicit blocking mode rejects only candidate-introduced degradation, after operators calibrate dream_adversarial_margin on their task mix and set dream_adversarial_rollouts to at least two.

Proof

The deterministic proof covers the pre-registered contract scenarios, each comparing the candidate arm versus the baseline arm on identical source and probe pairs.

1. A planted literal-only rule passes the ordinary held-out gate, is flagged by the probes, remains accepted in advisory mode, and is rejected in blocking mode; its evidence rows retain all four aggregated scores and show the gap change of -1.0 is candidate-introduced.
2. A backend that is equally frame-sensitive with and without the candidate produces a zero gap change and is never flagged, so pre-existing sensitivity cannot block adoption.
3. A candidate that improves both the source and probe scores while keeping the pre-existing gap produces a zero gap change and is never flagged, so a strictly better candidate cannot be rejected.
4. A single-rollout fluctuation under the candidate produces a negative mean gap change but only half the rollout indices worsen, so the majority rule keeps the row stable and the token cost of the extra rollout is the only effect.
5. A surface-robust candidate produces identical held-out gate results, artifacts, and acceptance decisions with probes disabled and enabled, and the fallback accounting shows zero probe influence on the training path.

Regression tests also pin train, validation, test, and provenance isolation; target-only routing in dual-backend operation; the three-variant, 256-probe, and eight-rollout resource bounds; strict configuration validation including the blocking rollout floor; non-finite and zero-probe fail-closed behavior; redaction; and Markdown-safe reporting.

These are deterministic contract tests, not a live-provider performance claim, so no stochastic lift or confidence interval is claimed.

Academic Support

• Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh, "Beyond Accuracy: Behavioral Testing of NLP Models with CheckList," ACL 2020, arXiv:2005.04118. CheckList motivates capability-focused behavioral tests and invariance checks instead of relying on aggregate accuracy alone; this design applies that principle to candidate-skill gating.
• Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits, "Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment," AAAI 2020, arXiv:1907.11932. TextFooler demonstrates that meaning-preserving surface changes can flip model behavior; this supports probing candidates with bounded request-frame variants.
• Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr, "Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting," ICLR 2024, arXiv:2310.11324. Its prompt-format sensitivity results motivate the wrapper, delimiter, and boundary perturbations exercised by these probes.
• Alexander Pan, Kush Bhatia, and Jacob Steinhardt, "The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models," ICLR 2022, arXiv:2201.03544. Its analysis of proxy misspecification supports candidate-specific stress tests, a baseline-relative attribution rule, and an advisory-first rollout before probe results can block adoption.

Testing

The focused integration slice contains 202 passing tests, 2 narrow optional skips, and 6 subtests on every native runner. It covers the planted brittle, equally sensitive, improves-both, single-fluctuation, and robust candidates; split and provenance isolation; cycle and multi-skill wiring; configuration failures including the blocking rollout floor; bounded expansion; evidence JSON with all four scores and per-rollout samples; diagnostics; Markdown reporting; staging; adoption; and target routing.

Platform	Python	Complete suite	Focused integration	Strict docs
Ubuntu	3.10	1,538 passed; 12 skipped; 353 subtests	202 passed; 2 skipped; 6 subtests	n/a
Ubuntu	3.11	1,538 passed; 12 skipped; 353 subtests	202 passed; 2 skipped; 6 subtests	n/a
Ubuntu	3.12	1,538 passed; 12 skipped; 353 subtests	202 passed; 2 skipped; 6 subtests	passed
macOS arm64	3.12	1,538 passed; 12 skipped; 353 subtests	202 passed; 2 skipped; 6 subtests	passed
Windows	3.12	1,493 passed; 57 skipped; 353 subtests	202 passed; 2 skipped; 6 subtests	passed

Every job asserted exact candidate 84bbde9ac19113bf4dee29dbfbf0755aacde6b36 before testing. No test was deselected and no failure was masked. The Windows difference is explicit platform and optional dependency skips, not failures.

Limitations & Negative Results

• These probes cover request-frame brittleness, not semantic correctness or a general adversarial guarantee; any fixed perturbation suite is an incomplete proxy.
• The aggregation rule is deliberately assumption-free (means plus sign-consistency across paired rollouts); it is a screening decision, not a significance test. Operators who want full inference can replay the retained per-rollout samples through the paired evalkit.
• No live paid-provider result is claimed; the committed evidence deterministically exercises the real gate, evidence, and report plumbing.
• Enabled probes replay each eligible source and probe pair once per rollout under both arms, so the replay cost is rollouts times two times the pair count; the factor, probe, and rollout caps bound it.
• Blocking remains opt-in because the margin should be calibrated on each operator's task distribution from advisory evidence first.

Reproduce It Yourself

python -m pytest tests/test_adversarial_dream.py -q
python -m pytest -q
python -m mkdocs build --strict

[Bogdan (Dan) Baciu (bogdanbaciu21)]

Ready for review.

This is one commit and nine files on the current base, at exact head f1b2c651781eaf1f80d307b727218b68a84d4f9c. The PR body now includes the design rationale, deterministic proof, four directly relevant academic references, exact native test receipts, limitations, and reproduction commands.

The focused integration slice passed on Ubuntu, macOS, and Windows: 177 passed, 2 optional skips, and 6 subtests on each runner. The complete suites also passed with zero failures: Ubuntu and macOS 1,513 passed; Windows 1,468 passed. Strict docs passed on Python 3.12.

The CLA check passed. The upstream CI run is marked action_required because the fork workflow is waiting for repository approval; no CI job failed.

[Yifan Yang (Yif-Yang)]

Thanks for the unusually thorough test matrix and for making the feature opt-in. The direction is useful, but the blocking decision is not yet sound enough to gate adoption.

The main issue is attribution. evaluate_adversarial_probes() evaluates only the candidate skill/memory and flags candidate_probe_score - candidate_source_score < -margin. It never evaluates the same source/probe pair against the baseline skill/memory. Consequently, blocking mode cannot distinguish brittleness introduced by the candidate from prompt-frame sensitivity already present in the backend. A candidate that materially improves both variants, or simply preserves an existing source/probe gap, can be rejected even though adopting it is strictly better.

Please make blocking baseline-relative: evaluate baseline and candidate on identical source/probe pairs, then base rejection on a paired change such as the candidate gap worsening relative to the baseline gap. Evidence should retain all four scores so the decision is auditable. At minimum, add regressions where (1) baseline and candidate are equally frame-sensitive and adoption is not blocked, (2) the candidate improves both scores but retains a gap and is not blocked, and (3) only candidate-introduced degradation is blocked.

There are two related validity problems:

• Each source and probe is currently evaluated once through replay_batch(). With a stochastic provider and the default margin=0, any one-sample fluctuation can mark a row brittle. Please use paired repeated rollouts with a documented aggregation/confidence rule, or keep this evidence strictly advisory until such a rule is implemented and calibrated.
• _strip_polite_frame() treats every leading Can/Could/Would you ... as removable boilerplate. That is not generally meaning-preserving—for example, Can you swim? becomes Swim?. Please restrict transformations to cases with a defensible semantic-preservation contract and add negative tests for ability/permission/question forms.

Until baseline-relative comparison, stochastic robustness, and conservative transformations are covered, please remove or disable the blocking path and keep the probes advisory-only. The deterministic mock tests and cross-platform green suite verify plumbing, but they do not validate the blocking signal itself.

[Bogdan (Dan) Baciu (bogdanbaciu21)]

Got it. Will do. These are incredibly helpful comments thank you so much for the time and care you took to provide them. It will take me a bit to make these changes, will reply when complete at a high quality level.

[Bogdan (Dan) Baciu (bogdanbaciu21)]

Thanks again for the review. All three issues are addressed at head 84bbde9ac19113bf4dee29dbfbf0755aacde6b36 (one new commit on top of the reviewed head, so the delta is inspectable).

1. Blocking is now baseline-relative. evaluate_adversarial_probes() scores identical source/probe pairs under both the current (baseline) documents and the candidate documents, and rejection is based on the paired change: a row is brittle only when the candidate gap worsens beyond the margin relative to the baseline gap. The baseline documents are required arguments so no caller can silently compare against an unintended baseline. Evidence rows retain all four aggregated scores plus the per-rollout samples. The three regressions you named are in: equal frame sensitivity in both arms is not blocked; a candidate that improves both scores but retains the gap is not blocked; only candidate-introduced degradation is blocked.

2. Stochastic validity: each task/arm is now replayed dream_adversarial_rollouts times (distinct sample ids so caching cannot collapse the samples), scores aggregate by mean, and brittleness additionally requires the worsening to hold in a strict majority of paired rollout indices. Blocking mode requires at least two rollouts (config validation rejects blocking at one), so a single-sample fluctuation can never reject a candidate; a regression pins exactly that case. The rule is documented in the docstring and docs as a screening decision, not a significance test; the retained per-rollout samples can be pushed through the paired evalkit for full inference, and the docs now say to calibrate the margin from advisory evidence before enabling blocking.

3. _strip_polite_frame now has an explicit semantic-preservation contract: only a leading "please" or a politeness-marked modal request ("can/could/would you please ...") is reframed; bare modal questions are never touched, and the first-person desire pattern ("I need/want you to ...") is removed entirely. Negative tests cover ability, permission, desire, and question forms ("Can you swim?", "Would you like some tea?", "I want you to know this matters."), plus positive tests for the politeness-marked forms.

Validation at the exact head across five native runners (Ubuntu 3.10/3.11/3.12, macOS arm64, Windows): full suite 1,538 passed on Ubuntu and macOS and 1,493 on Windows with zero failures; the focused slice is 202 passed everywhere; strict docs pass.

On your closing point: we considered removing the blocking path entirely and keeping probes advisory-only. We kept blocking opt-in behind the three conditions you set (baseline-relative comparison, repeated-rollout consistency, conservative transformations), plus the rollout floor, and advisory remains the default. If you would still prefer advisory-only until the margin has been calibrated on a public scenario, we are happy to disable blocking in this PR and propose it separately with that calibration.