Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 71 additions & 0 deletions BENCHMARKS.md
Original file line number Diff line number Diff line change
Expand Up @@ -1114,3 +1114,74 @@ fun (85 % of turns); the engine and the Python prototype end every individual in
adversaries or floors (a floor holds by construction under the soundness condition of docs/persona.md §2.9 and `prove` reports
it; this run measures it once); a bound on gaps when higher-priority habits preempt; anything about the words a model writes;
the times on another machine or an idle one.

## §10 Senses: individuality, habituation, replay and habits (`python3 bench/individuality/run.py --only a,b,d,e`, stdlib, 10.0 s; development build of 0.9.0)

The benchmark's spec, fixtures and bars are in [bench/individuality/README.md](bench/individuality/README.md); the fixtures and
bars were frozen with their sha256 (`bench/individuality/fixtures/MANIFEST`) before any run. One run on an Apple M4:

| metric | measured | bar | |
|---|---|---|---|
| a1 twins: share of 20 probes whose host action differs between an individual's two lives (40 individuals) | 0.145 (95 % interval 0.096 to 0.199) | >= 0.30, interval low >= 0.20 | fails |
| a2 twins: mean odds distance between two lives vs between two individuals in one room | 0.1307 vs 0.1329 (room A), 0.1411 (room B) | >= the smaller | fails |
| b1 habituation: zd at reading 20 / reading 1 (no genes); values against an independent Python sense step | 0.304149; 4,522 compared, 0 differ | <= 0.35; 0 | passes |
| b2 surprise: z after 40 quiet readings / after 40 at rate 0.3 | 6 / 1.75286 = 3.422977 | >= 2 | passes |
| b3 genes: individuals 0-99 whose zd at reading 20 is below reading 1 | 100 % | 100 % | passes |
| b4 named past: divergent decisions whose `why` clause equals its template rendered from the document | 100 % of 232 | >= 95 % | passes |
| b5 named past: divergent probes where a cited value differs between the two lives | 100 % of 116 | >= 95 % | passes |
| d replay: differing bytes, third and split copies of 80 lives; `live verify` failures (240 strands); the 0.8.2 strand | 0; 0; verifies, 20 events append the recorded bytes | 0; 0; equal | passes |
| e habits: violations and independent breaks over 9,442 stances; stances where a habit bound | 0 and 0; 72 | 0 and 0; >= 50 | passes |

Reading it. Habituation, surprise, the named past, replay and habits hold as specified. The twins bar does not: in this release
rewards pay the previous stance, and at these settings a different individual (another seed in the same room: 0.25 and 0.345
of probes) moves the host action more than a different life does (0.145). The bars are not changed after the fact; metric a is
re-measured when action-addressed feedback (RFC 0001 Design B, metric c) lands.

**P0** (`bench/individuality/p0.py check`, 77 s): 653 outputs of the released 0.8.2 binary over 8 personas without a senses
block (9,600 live turns and as many replayed, `describe`, `lint --seeds 0-99`, `fuzz` and `prove` documents): 0 differ. 629 are
byte-identical; 24 are the stderr of `live` runs whose scripts send a `senses` key, identical once the one line naming it is
removed. All 8 recorded strands verify, and the tutor's continues with the recorded bytes.

## §11 Action-addressed feedback: credit accuracy, and the twins re-measured (`python3 bench/individuality/run.py --only c,d,e`, `python3 bench/individuality/twins_addressed.py`, stdlib; development build of 0.9.0)

Metric c's fixtures, draws and bars were frozen in `bench/individuality/fixtures/MANIFEST` before it was ever run; three runs
gave the same values. Ten individuals of an act-or-ask policy (`fixtures/dispatch.yaml`: the shipped dispatch policy with a
credit block) make 300 decisions each; a host reads each stance's action, draws its outcome (good with probability 0.8 after
act, 0.4 after ask) and sends it back, on time (stream 1) or 0 to 6 events late with cancels, duplicates and payments of
turns never opened (stream 2). An independent Python learner replays RFC 0001 B.2 from the stance documents and the events.

| metric | measured | bar | |
|---|---|---|---|
| c1 share of the applied step equal to the learner's (and the same paid turn and eligibility), over 8,393 accepted payments in 50 runs | 100 % | >= 99 % | passes |
| c2 events whose Δ equals the learner's, after every one of 15,000 accepted events | 100 % | 100 % | passes |
| c3 outcomes accepted, of the 3,802 whose decision was open on arrival (streams 1 and 2, addressed) | 100 % | >= 99 % | passes |
| c4 duplicates, outcomes after a cancel and unknown turns (addressed) and ambiguous unaddressed feedback refused; runs equal to the run without their refused events | 100 % of 2,793; 50 of 50 byte-identical | 100 %; all | passes |

Also measured (not gated): the learner's open decisions equal the engine's `open_credit` after every event. Without the block
(0.8 semantics) 27 % of the outcomes credit a stance that took another level than the decision they were about, and all 138
duplicates and unknown payments move the weights; the 0.8 workaround (keep an outcome only when the previous stance took the
paid level) keeps 72 % of them. Act rate in the last 100 decisions: 0.655 on time, 0.628 addressed, 0.424 under 0.8
semantics, 0.657 with the workaround.

Reading it. Every accepted payment moved exactly the step RFC 0001 B.2 gives for the decision it named, and every duplicate,
cancelled or unknown payment was refused without changing anything. The stream is busier than its design intended: each
decision yields a message and an event carries at most one, so messages queue (254 still undelivered at the end over the
ten individuals) and with every event awaiting, 67 % of the addressed decisions were evicted at `max_open` 8 before their
outcome arrived (1,710 such outcomes refused, B-R2); paid outcomes were 0 to 7 events late. The fixture is not changed after
the fact.

**The twins re-measured with Design B** (`twins_addressed.py`, the frozen metric a fixture, streams, probes and bars; the
credit block and the addressed feedback added at run time: room A's praise names the decision 2 events earlier, room B's
criticism the one 3 events earlier):

| run | a1 mean d_life (95 % interval) | a2 tv_life vs min(tv_seed_A, tv_seed_B) | |
|---|---|---|---|
| as frozen (rewards pay the previous stance) | 0.145 (0.096 to 0.199) | 0.1307 vs 0.1329 | fails |
| addressed (Design B) | 0.1725 (0.1175 to 0.235) | 0.1223 vs 0.1366 | fails |
| control: readings as plain 0.8 number inputs against a fixed prior, rewards unaddressed (reported) | 0.155 (0.099 to 0.219) | 0.0295 vs 0.1296 | — |

Reading it. Addressing the rewards raises the share of probes whose action differs between two lives from 0.145 to 0.1725,
below the 0.30 bar. In both rooms the feedback still moves the learned caution weights away from `careful` (mean Δ careful
−0.32 after room A, −0.11 after room B), so the two lives drift the same way, and on the 10 probes with a tool failure the
habit and the failure's push make both lives careful or measured (23 of 400 actions differ). The control shows what the
normals add: with plain inputs the odds distance between two lives falls from 0.13 to 0.03.
30 changes: 30 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,36 @@
All notable changes to this project are documented here. The format follows
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/).

## 0.9.0 - unreleased

### Added
- **Senses** (docs/persona.md §2.11): an optional `senses` block. The host sends measured readings in declared units with
a named source; each is judged against a learned normal per sense (mean, variance, count; half-life in events; drift and
variance caps anchored to a declared prior) and moves moods and traits by its capped z-score; `why` names the normal and
what the learned weights came from. Readings from `self` or through MCP are refused. A persona without the block gives
byte-identical documents.
- `live verify` summarises the normals (`normals`: mean, count and the largest move within 50 events); `monitor` draws a row
per sense and an alert when a normal moved over `drift_cap / 2` within 50 events. `examples/persona/sensed.yaml` (with its
golden and rules) and `bench/individuality/` (benchmark metrics a, b, d and e; the P0 corpus) are new. Design notes:
`docs/rfcs/0001-senses-and-reward.md`.
- **Action-addressed feedback** (docs/persona.md §2.8): an optional `learning.credit` block. An event can mark its decision
as awaiting an outcome (`await`), a reward can name the decision it pays for (`for: <turn>`), and decisions can be
cancelled (`cancel`). Credit goes to the paid decision's own odds, scaled by an eligibility that halves per
`half_life_events` of delay; at most `max_open` decisions wait; ambiguous, duplicate, cancelled and unknown payments are
refused whole. Without `for`, feedback pays the previous stance as before.
- `for: {turn, stance}` binds the paid stance's digest (a mismatch is refused); every stance with a credit block carries
`credit` (the payment, the decisions closed, the ones still open, its own digest). A control line closes the open
decisions. `live verify` adds `open` to its summary, and `live verify --actions FILE` compares a host's action log with
the released levels; `monitor` draws the open decisions with their eligibility; `fuzz` adds late, reordered, duplicated,
cancelled and unknown payments; the MCP turn tools refuse `cancel`. `examples/agent-harness/addressed.json` (with
`late.py` and its golden) and benchmark metric c are new.

### Changed
- `live` says once on stderr when a persona without a senses block receives a `senses` key (still an undeclared input,
listed in `ignored`). The monitor's panel of the event's inputs is labelled INPUTS (it was SENSES).
- `live` says once on stderr when a persona without a credit block receives `await`, `for` or `cancel` (still undeclared
inputs, listed in `ignored`; feedback pays the previous stance).

## 0.8.3 - unreleased

### Added
Expand Down
2 changes: 2 additions & 0 deletions bench/individuality/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
result.json
__pycache__/
Loading
Loading