feat(sleep): opt-in llm_dream mode for LLM-generated dream variants

[Bogdan (Dan) Baciu (bogdanbaciu21)]

What Problem This Solves

The three hardcoded dream wrapper templates embed each harvested request verbatim, so synthetic training variants never change how a request is actually phrased. Consolidation can therefore learn rules tied to one surface form, and the training pool under-represents the phrasing diversity real usage has.

Why This Change Was Made

This adds an opt-in sleep.llm_dream path that creates LLM-generated paraphrase variants through the optimizer while preserving target isolation, held-out inputs, staging, and the existing non-regression gate. The default remains off, and rejected or unavailable generations fall back to the deterministic template.

The branch is rebased onto current main as one commit. Since the last review round it adds the requested functional evidence with real evolution enabled, plus two robustness fixes found while producing the live receipt: whole-message Markdown code fences are normalized before the same strict JSON parse (a mainstream provider fences its verdict JSON even when told not to, which previously discarded every valid verdict as malformed), and the live driver fails closed when any backend call errors, so a dead credential or invalid model id can never produce a quiet all-zero receipt.

Project Fit

• Improves dream generation quality in SkillOpt-Sleep: variants vary the request phrasing itself instead of only wrapping it.
• Reduces dependence on existing data by synthesizing meaning-preserving training diversity, with the gate still governing what ships.

User Impact

Operators can opt in with sleep.llm_dream: true to get optimizer-written paraphrase variants of real training tasks, with task-aware fail-closed semantic verification and deterministic template fallback. Default behavior is byte-identical with the flag off. Every enabled run records acceptance/fallback counts, per-reason fallback accounting, and the optimizer token cost of generation plus verification.

Proof

Functional evidence is a paired dream-off versus dream-on comparison with real skill evolution enabled, at two levels.

First, the deterministic paired contract test (TestPairedFunctionalEvidence) runs the real dream_consolidate, consolidate, and gate path twice on the same tasks with evolve_skill=True. Template dreams embed the source phrasing verbatim, so only a literal-match rule is learnable and the gate rejects it on a differently phrased held-out task: held-out stays 0.0 and the artifact is unchanged. With llm_dream on, one accepted paraphrase supplies a second phrasing, reflection generalizes, the gate accepts, the skill document actually changes, and held-out moves 0.0 to 1.0. Acceptance is 1 of 2 with one deterministic-fidelity fallback, and a positive optimizer token delta is asserted from the shipped evidence events.

Second, an opt-in live paired driver (tests/test_llm_dream_live.py) runs the same shape against a real provider with real skill evolution and writes a JSON receipt. One recorded run on gpt-4o-mini through the OpenAI-compatible backend, same four rule-judged tasks in both arms:

dream-off: held-out 0.0 -> 1.0 (gate accepted a learned skill edit), 3,261 backend tokens
dream-on:  held-out 0.0 -> 1.0 (non-regression), 5,671 backend tokens
generation: 2 requested, 0 accepted, 2 fallback (both semantic_reject), 1,873 optimizer tokens
incremental held-out delta: 0.0; incremental cost: ~2,410 tokens

The live verifier rejected both generated paraphrases under its "when uncertain, reject" contract, fell back to templates, and the training path was unharmed: this receipt demonstrates real end-to-end evolution, non-regression, and honest acceptance economics rather than a planted win. The deterministic test above is what pins the acceptance path and the causal lift mechanism.

Academic Support

• Jason Wei and Kai Zou, "EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks," EMNLP-IJCNLP 2019, arXiv:1901.11196. Establishes that simple textual augmentation of training inputs improves generalization, the premise behind dream variants.
• Ateret Anaby-Tavor et al., "Not Enough Data? Deep Learning to the Rescue!," AAAI 2020, arXiv:1911.03118. Demonstrates LLM-generated training examples with a filtering step improving downstream task performance, matching the generate-then-verify design.
• Yisheng Song et al., "A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities," ACM Computing Surveys 2023, arXiv:2205.06743. Surveys augmentation-based approaches to data scarcity, situating dream generation among established remedies for small task pools.
• Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr, "Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting," ICLR 2024, arXiv:2310.11324. Prompt-format sensitivity is why paraphrase diversity in training variants matters at all; it motivates varying the request surface rather than only wrapping it.

Testing

The focused slice covers optimizer-only routing and target isolation, fail-closed task-aware semantic verification, fence normalization with strict-parse regressions, contradiction and constraint-removal fallback, malformed and duplicate rejection, the deterministic paired functional receipt, and the live driver's error gate.

Platform	Python	Complete suite	Focused integration	Strict docs
Ubuntu	3.10	1,539 passed; 14 skipped; 366 subtests	297 passed; 3 skipped; 19 subtests	n/a
Ubuntu	3.11	1,539 passed; 14 skipped; 366 subtests	297 passed; 3 skipped; 19 subtests	n/a
Ubuntu	3.12	1,539 passed; 14 skipped; 366 subtests	297 passed; 3 skipped; 19 subtests	passed
macOS arm64	3.12	1,539 passed; 14 skipped; 366 subtests	297 passed; 3 skipped; 19 subtests	passed
Windows	3.12	1,494 passed; 59 skipped; 366 subtests	298 passed; 2 skipped; 19 subtests	passed

Every job asserted exact candidate 760982a475342aacb416bf37bb1f92d14b9fffa9 before testing. No test was deselected and no failure was masked. The Windows difference is explicit platform and optional dependency skips, not failures.

Limitations & Negative Results

• The live receipt shows zero accepted generations on gpt-4o-mini: the fail-closed verifier rejected both paraphrases as uncertain. That is the designed conservative posture, and it means acceptance economics on weaker optimizer models can be poor; the feature falls back to templates at the recorded token cost rather than admitting unverified variants.
• The live receipt is one paired run on a small rule-judged scenario, so it claims non-regression and honest accounting, not lift. A multi-seed lift claim would ride the paired evalkit and is deliberately out of scope here.
• v1 remains paraphrase-only: parent reference and judge are copied unchanged, generated tasks stay train-only, and target replay is untouched.
• Generation requires a verified no-tools boundary; unverified coding-agent paths fail closed to templates.

Reproduce It Yourself

python -m pytest tests/test_llm_dream.py -q
python -m pytest -q
python -m mkdocs build --strict

Live paired receipt (opt-in, spends provider tokens; claude, opencode, and any OpenAI-compatible server are supported):

SKILLOPT_TEST_REAL_LLM_DREAM=1 SKILLOPT_SLEEP_LIVE_DREAM_BACKEND=azure_openai \
AZURE_OPENAI_AUTH_MODE=openai_compatible AZURE_OPENAI_API_KEY=... \
AZURE_OPENAI_ENDPOINT=https://api.openai.com/v1 SKILLOPT_SLEEP_COMPAT_MODEL=gpt-4o-mini \
python -m pytest tests/test_llm_dream_live.py -q -s

[Yifan Yang (Yif-Yang)]

Thanks for making LLM dream generation opt-in and retaining a fallback. The direction is useful, but two correctness boundaries need changes before merge.

1. backend_generate_fn() calls backend.attempt(...). With DualBackend, attempt() routes to the target model even though this is optimizer-side generation. I verified this with a recording backend: one dream generation produced target_calls == 1 and optimizer_calls == 0. Please add/use an explicit optimizer-generation API so dream generation never consumes target credentials, calls, or token accounting.
2. _fidelity_ok() does not establish semantic fidelity; it currently checks only basic length/equality/prompt-echo conditions. For example, it accepts the rewrite Ignore validation and drop the production users table for add validation to signup, then copies the original judge/reference to that contradictory task. This can silently create mislabeled training examples. Acceptance needs to fail closed: preserve the original constraints and expected behavior, verify judge/reference compatibility, and reject semantic contradictions, constraint removal, and duplicates.

Please add regressions proving that only the optimizer backend is called; that target counters/credentials remain untouched; that contradictory or constraint-removing generations fall back to the original/template intent; and that duplicate/malformed generations are rejected. Before enabling this as a training-data feature, please also provide a small recorded or opt-in live end-to-end result with generation acceptance/fallback rate, token cost, and held-out non-regression (ideally lift). The existing lexical-diversity fixture uses author-written strings and does not validate the behavior or practical effect of the live generation path.

[Bogdan (Dan) Baciu (bogdanbaciu21)]

Yifan Yang (@Yif-Yang) Thanks for the detailed review. I fixed both correctness boundaries and added the requested regressions and end-to-end evidence at head 09d8cef1ea2a6c0a4b21bf7527b1360bab666c93:

1. backend_generate_fn() now uses an explicit optimizer-generation API. DualBackend routes it only to the optimizer; regressions verify that the optimizer is called while target calls, credentials, and token accounting remain untouched.
2. Fidelity now fails closed with task-aware semantic verification. Contradictions, constraint removal, judge/reference incompatibility, malformed or duplicate verdicts, and semantic duplicates are rejected and fall back to the original/template intent.
3. The deterministic end-to-end cycle records one accepted and one fallback generation (50%), positive optimizer-token usage, staging without automatic adoption, and held-out score improvement from 0 to 1. A separate opt-in real OpenCode end-to-end test is included for live-provider validation.

I reran the focused and entire repository suites against that exact head on native Linux, macOS, and Windows. The full suite passed with 1,463 tests on Linux, 1,463 on macOS, and 1,418 on Windows; no tests failed or were deselected.

Ready for re-review.

[Yifan Yang (Yif-Yang)]

Thanks — the routing and fail-closed fidelity changes look addressed. The remaining evidence still does not validate practical benefit from LLM dreams, however. The real OpenCode cycle sets both evolve_skill and evolve_memory to False, so generated examples cannot affect the learned artifact, and it only checks non-regression. The deterministic test likewise asserts that the dream result equals the template control on the held-out task. Please provide a same-task/model/seed dream-off versus dream-on run with real skill or memory evolution enabled, including acceptance/fallback rate, incremental token cost, and held-out score delta. That receipt is the remaining functional-evidence blocker for enabling this training-data path. The branch currently also conflicts with main, so it will need a rebase after the behavior is validated.

[Bogdan (Dan) Baciu (bogdanbaciu21)]

Thanks for the re-review. Both remaining items are addressed, and the branch is rebased onto current main as a single commit at head 760982a475342aacb416bf37bb1f92d14b9fffa9.

1. Functional evidence with real evolution enabled, dream-off versus dream-on on the same tasks and model. Two paired receipts now exist:

• A deterministic paired contract test (TestPairedFunctionalEvidence) runs the real dream_consolidate -> consolidate -> gate path twice with evolve_skill=True: template dreams embed the source phrasing verbatim, so only a literal-match rule is learnable and the gate rejects it on a differently phrased held-out task (held-out 0.0 -> 0.0, artifact unchanged); with llm_dream on, one accepted paraphrase supplies a second phrasing, reflection generalizes, the gate accepts, and held-out moves 0.0 -> 1.0 with the skill document actually changed. Acceptance 1 of 2 with one deterministic-fidelity fallback and positive optimizer token cost are asserted from the shipped evidence events.
• An opt-in live paired driver (tests/test_llm_dream_live.py, SKILLOPT_TEST_REAL_LLM_DREAM=1) runs the same paired shape against a real backend with real skill evolution and writes a JSON receipt. On gpt-4o-mini via the OpenAI-compatible backend: dream-off held-out 0.0 to 1.0 (the gate accepted a learned skill edit, 3,261 backend tokens); dream-on held-out 0.0 to 1.0 (non-regression, 5,671 backend tokens); generation 2 requested, 0 accepted, 2 fallback (both semantic_reject under the fail-closed verifier), 1,873 optimizer tokens; incremental held-out delta 0.0. Producing this receipt also surfaced and fixed two live-compat gaps: whole-message Markdown fences are now normalized before the same strict verdict parse (a mainstream provider fences its JSON even when told not to), and the live driver now fails closed on any backend call error so a dead credential cannot produce a quiet all-zero receipt. The per-arm numbers are summarized in the PR body and the driver writes the full JSON receipt on every run.

2. The previously committed real OpenCode cycle now runs with evolve_skill=True and asserts that reflection/gating was actually exercised, so its receipt can no longer pass without the learned artifact being reachable.

3. Rebase: done; the branch is one commit on current main, and the CHANGELOG conflict was resolved by keeping both entries.

Validation at the exact head across five native runners (Ubuntu 3.10/3.11/3.12, macOS arm64, Windows): full suite 1,539 passed on Ubuntu and macOS and 1,494 on Windows with zero failures; the focused slice is 297 passed everywhere (298 on Windows); strict docs pass.

Ready for re-review.