diff --git a/.claude/blackboard.md b/.claude/blackboard.md index 07c79c65..126356ad 100644 --- a/.claude/blackboard.md +++ b/.claude/blackboard.md @@ -3967,7 +3967,7 @@ in bit-sliced counters. | # | fruit | size | gate | |---|---|---|---| -| U1 | wire `sum8_u8_add_masked` into candidate E (Esum's anchor term) and re-measure E against B | low | E's oracle tests unchanged; one row per tier | +| U1 | ~~wire `sum8_u8_add_masked` into candidate E (Esum's anchor term) and re-measure E against B~~ DONE: #393 (D-U9-E5; ADOPT-LAB only without VNNI) | low | E's oracle tests unchanged; one row per tier | | U2 | AVX2 streamed B is 27 - 33 ns against 9 - 10 in L1, more than the extra bytes explain | low | one probe that separates bandwidth from the masked byte-dot's 4 `vpshufb` | | U3 | NEON and wasm timings for B, D and F (only x86 measured) | low | a real aarch64 host; qemu numbers are not valid | | U4 | sign plane to `+1/-1` bytes without `mask_blend` (E3's cost on AVX2) | high | facade word with parity on all realizations; must pass the facet-stride guard | @@ -3975,3 +3975,13 @@ in bit-sliced counters. | U6 | `U8x64 -> I8x64` bit view for B's `low7`/`top` (today a store/load) | heavy | needs an operator ruling: facet-stride R3 forbids u8/i8 mixing | | U7 | recall on a real TurboVec-U9 index (arithmetic parity says nothing about retrieval) | heavy | owner experiment, defined in the head-to-head report | +## D-U9-E5: the U9 anchor through vertical masked byte sums (2026-10-29, branch claude/u9-e5) + +Probe-only: `AccE5` in `examples/u9_abacus/anchor.rs` (E3/E4/Esum/B unchanged), 5 tests in `tests/u9_abacus.rs`, +report `.claude/knowledge/u9-e5-probe.md`. Consumer gate for #390: E's split `|b| = low7 + 128*bit7`, two masked low7 +byte-dots, anchor as two persistent `U64x8::sum8_u8_add_masked` accumulators reduced only at flush. Exact on 7 +realizations; wasm/neon parity pass; 6 disable runs red. Verdict: ADOPT-LAB on AVX-512 without VNNI (L1 e2e 5.14-5.30 vs B +5.67-7.50, 8 runs; byte-dot emulated in ~10 insns there), REFUSE on VNNI (3.55-3.95 vs 2.92-3.25: B's anchor dot shares the +sign's zero-masked copy, E5 pays maskz+vpsadbw+vpaddq per sign; 0 spills) and AVX2 (mask built in registers per call, +8 vpshufb + 16 vpmuludq per block). Overall DEFER as B replacement. Closes fruit U1 (blackboard U9 fruit list, #392). + diff --git a/.claude/knowledge/u9-e5-probe.md b/.claude/knowledge/u9-e5-probe.md new file mode 100644 index 00000000..b441d0e9 --- /dev/null +++ b/.claude/knowledge/u9-e5-probe.md @@ -0,0 +1,185 @@ +# D-U9-E5: the U9 power-of-two anchor through vertical masked byte sums + +READ BY: savant-architect, simd-savant, vector-synthesis, the TurboVec-U9 owner. +Follows `u9-abacus-probe.md` (#389, candidate E) and the facade word +`U64x8::sum8_u8_add_masked` (#390). + +Scope: one new consumer candidate, `AccE5` in `examples/u9_abacus/anchor.rs`. +E3, E4, Esum and B are unchanged, so every earlier row stays reproducible. No +facade, backend, codec or storage change. + +## Answer + +**Does the vertical `U64x8` accumulator make the U9 anchor cheaper than B's +fourth byte-dot? Only where the byte-dot is not a single instruction.** + +| tier | verdict | complete-consumer result against B | +|---|---|---| +| AVX-512 without VNNI (v4) | **ADOPT-LAB** | L1 end-to-end 5.14 - 5.30 vs 5.67 - 7.50 ns per 64 (about 9 % faster, 8 runs, ranges disjoint); arithmetic 4.47 vs 5.08 typically, lower in all 8 paired runs; scan 4.67 vs 4.83; streamed end-to-end lower in every paired run | +| AVX-512 VNNI | **REFUSE** as a B replacement | L1 end-to-end 3.55 - 3.95 vs 2.92 - 3.25 (1.2x slower); scan 3.31 vs 2.62 - 3.05; streamed end-to-end tie (5.55 - 5.76 vs 5.50 - 6.11) | +| AVX2 (v3) | **REFUSE** as a B replacement | L1 end-to-end 12.2 vs 10.0; streamed end-to-end 40.0 - 40.5 vs 27.1 - 31.7 | + +Overall: **DEFER** as a general B replacement, **ADOPT-LAB** on AVX-512 without +VNNI. E5 is, on every tier, the best of the four E variants, and the first E +variant that beats B anywhere. + +## What E5 does + +E5 keeps E's split `|b| = low7 + 128 * bit7` and the two masked `low7` +byte-dots. The anchor term goes into two persistent vertical accumulators: + +```rust +self.anchor[0] = self.anchor[0].sum8_u8_add_masked(a, !negative & top); +self.anchor[1] = self.anchor[1].sum8_u8_add_masked(a, negative & top); +``` + +`top` is the bit-7 plane of `|b|` and `negative = plane_a ^ plane_b`. At flush: + +`total += (low7_pos - low7_neg) + ((reduce_sum(anchor_pos) - reduce_sum(anchor_neg)) << 7)` + +The anchors are reduced only at flush, every `B_FLUSH_BLOCKS` = 16,384 blocks, +never per block. `AccE5::reductions` counts the reductions, so a premature +reduction is observable. + +Bounds per flush: + +- a `low7` lane gains at most `4 * 255 * 127 = 129,540` per block, at most + 2,122,383,360 per flush, below `i32::MAX`; +- an anchor lane gains at most `8 * 255 = 2,040` per block, at most 33,423,360 + per flush. The 8 lanes sum to at most 267,386,880 and `<< 7` gives at most + 3.4e10, inside `i64`. + +## Measurements + +ns per 64 values, compact operand layout (72 bytes per block, as in #388 and +#389), mask construction inside every timed row. VNNI and AVX2 are 3 runs each; +v4 is 8 runs (3, then 5 more to confirm the win). Ranges are min - max of +medians. L1 = 64 blocks, streamed = 2^20 blocks. + +| L1 | AVX-512 VNNI | AVX-512, no VNNI | AVX2 | +|---|---|---|---| +| B end-to-end | 2.92 - 3.25 | 5.67 - 7.50 | 10.0 | +| E3 end-to-end | 2.94 - 3.36 | 5.42 - 6.59 | 38.1 - 53.8 | +| E4 end-to-end | 3.59 - 4.11 | 6.36 - 7.78 | 14.5 | +| Esum end-to-end | 4.45 - 4.75 | 5.98 - 6.56 | 60.0 - 64.6 | +| **E5 end-to-end** | 3.55 - 3.95 | **5.14 - 5.30** | 12.2 | +| B arithmetic only | 2.44 - 2.69 | 5.02 - 5.64 | 9.19 - 10.8 | +| E3 arithmetic only | 2.31 - 2.47 | 4.98 - 5.88 | 37.9 - 40.0 | +| **E5 arithmetic only** | 2.72 - 2.95 | **4.47 - 5.17** | 10.8 - 11.0 | +| B scan | 2.62 - 3.05 | 4.83 - 4.97 | 9.11 - 9.17 | +| E3 scan | 2.67 - 3.05 | 4.98 - 7.89 | 38.4 - 48.2 | +| **E5 scan** | 3.31 | **4.67 - 8.39** | 11.3 - 11.6 | + +| Streamed | AVX-512 VNNI | AVX-512, no VNNI | AVX2 | +|---|---|---|---| +| B end-to-end | 5.50 - 6.11 | 8.38 - 14.2 | 27.1 - 31.7 | +| E3 end-to-end | 5.31 - 5.38 | 7.01 - 12.6 | 216 - 223 | +| **E5 end-to-end** | 5.55 - 5.76 | **6.80 - 12.1** | 40.0 - 40.5 | +| B scan | 2.82 - 3.31 | 4.61 - 8.91 | 26.6 - 27.5 | +| E3 scan | 2.74 - 2.80 | 4.88 - 5.99 | 212 - 225 | +| **E5 scan** | 3.19 - 3.33 | **4.64 - 5.13** | 25.1 - 28.7 | + +E4 and Esum streamed are in the probe output and lose to E5 on every tier. The +streamed arithmetic-only rows read precomputed operands, so they measure +memory, not arithmetic, and are left out, as in #389. + +Notes: + +- v4 L1 scan: 7 of 8 runs were 4.67 against B's 4.83; one run read 8.39, + noisy on that run only. +- v4 streamed end-to-end: the ranges overlap, because the host varies between + runs. Paired by run (5 confirmation runs), E5 is lower than B every time: + 12.1/14.2, 10.2/12.3, 10.6/12.7, 7.8/11.9, 7.3/10.4. + +### Flush overhead (ns per flush) + +| | VNNI | v4 | AVX2 | +|---|---|---|---| +| B: 4 x `I32x16` -> `i64` | 48.3 - 48.9 | 48.2 - 72.7 | 27.1 - 28.2 | +| E3: 3 x `I32x16` | 36.9 - 37.1 | 37.0 - 37.4 | 19.8 - 19.9 | +| E5: 2 x `I32x16` + 2 x `U64x8::reduce_sum` | 27.5 - 27.7 | 27.3 - 30.2 | 15.3 - 15.8 | + +The flush is cheaper for E5, but amortized over 16,384 blocks it is below +0.003 ns per 64 for every candidate, so it decides nothing. + +## Assembly (steady-state loop body, `#[inline(never)]` loop copies) + +| tier | B body | E5 body | spills in the body | E5 per block | +|---|---|---|---|---| +| VNNI | 18 instructions: 4 `vpdpbusd`, 2 `vmovdqu8 {z}` | 23: 2 `vpdpbusd`, 4 `vmovdqu8 {z}`, 2 `vpsadbw`, 2 `vpaddq`, `kandq`, `kandnq` | 0 / 0 | no shuffle, no horizontal reduction | +| v4 (no VNNI) | 40: 8 `vpmaddwd`, 8 `vpaddd`, shifts | 36: 4 `vpmaddwd`, 4 `vpaddd`, 2 `vpsadbw`, 2 `vpaddq`, 4 `vmovdqu8 {z}` | 0 / 0 | no shuffle, no horizontal reduction | +| AVX2 | 154, 46 stack operands | 185, 42 stack operands: 8 `vpshufb`, 16 `vpmuludq`, 8 `vpcmpeqb` | yes / yes (array-backed AVX2 types) | the keep mask is built in registers per call | + +At flush, LLVM folds `reduce_sum(pos) - reduce_sum(neg)` into one `vpsubq` +followed by one horizontal reduction (`vextracti64x4`, `vextracti128`, +`vpshufd`). That runs once per 16,384 blocks. + +## Why E5 loses where it loses + +- **AVX-512 VNNI: mask expansion.** The cause is not register pressure (0 + spills), not the two extra accumulator chains (they are independent and the + loop has no carried dependency beyond the adds), and not memory traffic + (streamed, E5 ties B). B's fourth byte-dot is one `vpdpbusd` that reuses the + zero-masked copy of `a` the sign already needs: both of B's dots per sign + share one mask. E5's anchor mask is `sign AND bit7`, which differs from the + `low7` mask, so every anchor term pays its own zero-masked move, a `vpsadbw` + and a `vpaddq`. That is three instructions against one, per sign. +- **AVX2: mask expansion again.** `sum8_u8_add_masked` builds its keep mask in + registers (variable shift, multiply, compare) on every call. That costs more + than the byte-dot it replaces. Both kernels spill because the AVX2 `U8x64` + and `I32x16` are array-backed; E5 spills slightly less. + +## Why E5 wins on AVX-512 without VNNI + +Without VNNI, a masked byte-dot is emulated with about 10 instructions +(`vpmaddwd`, `vpaddd` and shifts). E5 replaces two of B's four byte-dots with +two SAD sums. A SAD sum costs 3 instructions including its own mask, so the +mask overhead that sinks E5 on VNNI is smaller than the work it saves. + +Conditions for the win: + +- AVX-512BW without VNNI; +- the compact 72-byte block layout; +- the `b` side either prepared per block (end-to-end) or once as a query (scan). + +The win holds in L1, is smaller in the scan (3 %), and streamed it is lower in +every paired run but within host noise. + +## Correctness + +`tests/u9_abacus.rs`, 5 new tests (20 in the file), against #388's `i64` +oracle and against B: + +- all 511 x 511 numeric pairs, one lane each, end-to-end and as a scan; +- mixed full blocks of 1, 2, 7, 64 and 300 blocks with random signs. The test + asserts that mixed signs and bit-7 magnitudes are present; +- all-255 runs past three flush bounds with all four sign mixes; totals beyond + `i32`; +- one anchor reduction per flush (`AccE5::reductions`); +- touched zero scores 0, and never-touched is refused before E5 sees the block. + +The file passes on native (AVX-512 VNNI), v4, v3, AVX (qemu SandyBridge), v2 +(qemu Nehalem), aarch64 (qemu) and nightly-simd. `scripts/wasm-parity.sh` and +`scripts/neon-parity.sh` pass; the backends are unchanged. Clippy `-D warnings` +(native, v3) and rustfmt are clean. The facet-stride guard reports nothing on +the added lines. + +Disable runs, each red: + +| id | disable | result | +|---|---|---| +| X1 | bit-7 gating lost on the positive anchor (`!negative` instead of `!negative & top`) | RED (3 tests) | +| X2 | polarity inverted (positive and negative anchor masks swapped) | RED (3 tests) | +| X3 | reconstruction `<< 6` instead of `<< 7` | RED (3 tests) | +| X4 | anchor reduction missing at flush | RED (3 tests) | +| X5 | anchors not reset at flush (double count) | RED (1 test, the run across three flush bounds) | +| X6 | premature reduction, flushing every block | RED (1 test, the reduction count). Every exactness test stays green: an early reduction is a cost defect, not a wrong result, so it needs a counter to be visible | + +## Not done + +- NEON and wasm timings. The parity harnesses pass; qemu and node timings are + not meaningful. +- An E5 variant whose anchor mask is shared with the `low7` mask. It would + need a SAD word that takes an already-masked vector plus a second mask, or a + different split. Not built; it would be a new facade word. +- Integration anywhere outside the probe. diff --git a/examples/u9_abacus/anchor.rs b/examples/u9_abacus/anchor.rs index 895eef45..b7142256 100644 --- a/examples/u9_abacus/anchor.rs +++ b/examples/u9_abacus/anchor.rs @@ -14,14 +14,16 @@ //! * E3 ([`AccE3`]): one byte-dot of `a` against a `+1 / -1` byte vector //! (built from the sign plane), masked by the bit-7 plane: three byte-dots; //! * Esum ([`AccESum`]): `U8x64::mask_blend` + `sum_bytes_u64` per sign, which -//! reduces horizontally on every call. +//! reduces horizontally on every call; +//! * E5 ([`AccE5`]): `U64x8::sum8_u8_add_masked` (#390) per sign into two +//! persistent vertical accumulators, reduced only at flush. //! //! The polarity rule is unchanged: `negative = plane_a XOR plane_b`. #![allow(dead_code)] use crate::kernels::{bits_as_i8, lanes_i64, Block, B_FLUSH_BLOCKS}; -use ndarray::simd::{I32x16, I8x64, U8x64}; +use ndarray::simd::{I32x16, I8x64, U64x8, U8x64}; /// Lanes whose magnitude has bit 7 set, as a plane. #[inline(always)] @@ -211,6 +213,74 @@ impl AccESum { } } +/// E5: the two `low7` byte-dots, and the anchor sums as two persistent +/// vertical masked byte sums (`U64x8::sum8_u8_add_masked`), reduced only at +/// flush. +/// +/// Bounds per block: a `low7` lane gains at most `4 * 255 * 127 = 129_540`, +/// so after `B_FLUSH_BLOCKS` = 16_384 blocks it holds at most 2_122_383_360, +/// below `i32::MAX`. An anchor lane gains at most `8 * 255 = 2_040`, at most +/// 33_423_360 per flush; eight lanes sum to at most 267_386_880 and `<< 7` +/// to 3.4e10, inside `i64`. The `u64` lanes cannot overflow before the flush. +pub struct AccE5 { + low7: [I32x16; 2], + anchor: [U64x8; 2], + blocks: usize, + total: i64, + /// Number of anchor reductions so far; one per flush, never per block. + pub reductions: usize, +} + +impl AccE5 { + pub fn new() -> Self { + Self { + low7: [I32x16::splat(0); 2], + anchor: [U64x8::splat(0); 2], + blocks: 0, + total: 0, + reductions: 0, + } + } + + pub fn flush(&mut self) { + let r = self.low7.map(lanes_i64); + let anchor = self.anchor[0].reduce_sum() as i64 - self.anchor[1].reduce_sum() as i64; + self.total += (r[0] - r[1]) + (anchor << 7); + self.low7 = [I32x16::splat(0); 2]; + self.anchor = [U64x8::splat(0); 2]; + self.blocks = 0; + self.reductions += 1; + } + + #[inline(always)] + pub fn mac(&mut self, a: U8x64, negative: u64, low7: I8x64, top: u64) { + self.low7[0] = self.low7[0].dot4_u8_i8_add_masked(a, !negative, low7); + self.low7[1] = self.low7[1].dot4_u8_i8_add_masked(a, negative, low7); + self.anchor[0] = self.anchor[0].sum8_u8_add_masked(a, !negative & top); + self.anchor[1] = self.anchor[1].sum8_u8_add_masked(a, negative & top); + self.blocks += 1; + if self.blocks == B_FLUSH_BLOCKS { + self.flush(); + } + } + + #[inline(always)] + pub fn step(&mut self, a: Block, b: Block) { + let (low7, top) = split_anchor(b.mag); + self.mac(a.mag, a.positive ^ b.positive, low7, top); + } + + #[inline(always)] + pub fn scan_step(&mut self, x: Block, q: &QueryE) { + self.mac(x.mag, x.positive ^ q.positive, q.low7, q.top); + } + + pub fn finish(mut self) -> i64 { + self.flush(); + self.total + } +} + /// A query prepared once: `low7` bytes, bit-7 plane, polarity. pub struct QueryE { pub low7: I8x64, diff --git a/examples/u9_abacus/main.rs b/examples/u9_abacus/main.rs index 2050d129..bc45cb68 100644 --- a/examples/u9_abacus/main.rs +++ b/examples/u9_abacus/main.rs @@ -426,6 +426,11 @@ fn bench_e(label: &str, n: usize, runs: usize, seed: u64) { let (e4_e2e, e4_ari, e4_scan) = e_rows!(AccE4); let (e3_e2e, e3_ari, e3_scan) = e_rows!(AccE3); let (es_e2e, es_ari, es_scan) = e_rows!(AccESum); + let (e5_e2e, e5_ari, e5_scan) = e_rows!(AccE5); + // The out-of-line arithmetic loops read for the disassembly must also be exact. + assert_eq!(asm_b_loop(&ab.mag, &split), want); + assert_eq!(asm_e3_loop(&ab.mag, &anc), want); + assert_eq!(asm_e5_loop(&ab.mag, &anc), want); assert_eq!(b_e2e(), want); assert_eq!(b_ari(), want); assert_eq!(b_scan(), want_scan); @@ -435,14 +440,50 @@ fn bench_e(label: &str, n: usize, runs: usize, seed: u64) { time("E4 end-to-end: 2 dot4 low7 + 2 dot4 x ones, << 7", n, runs, Some(be), &e4_e2e); time("E3 end-to-end: 2 dot4 low7 + 1 dot4 x signed ones", n, runs, Some(be), &e3_e2e); time("Esum end-to-end: 2 dot4 low7 + 2 masked sum_bytes", n, runs, Some(be), &es_e2e); + time("E5 end-to-end: 2 dot4 low7 + 2 vertical masked sum8", n, runs, Some(be), &e5_e2e); let ba = time("B arithmetic only", n, runs, None, &b_ari); time("E4 arithmetic only", n, runs, Some(ba), &e4_ari); time("E3 arithmetic only", n, runs, Some(ba), &e3_ari); time("Esum arithmetic only", n, runs, Some(ba), &es_ari); + time("E5 arithmetic only", n, runs, Some(ba), &e5_ari); let bs = time("B scan", n, runs, None, &b_scan); time("E4 scan", n, runs, Some(bs), &e4_scan); time("E3 scan", n, runs, Some(bs), &e3_scan); time("Esum scan", n, runs, Some(bs), &es_scan); + time("E5 scan", n, runs, Some(bs), &e5_scan); +} + +/// Cost of one flush of an accumulator, ns per flush (reduction, `i64` +/// combine and reset). Amortized over `B_FLUSH_BLOCKS` blocks. +fn flush_costs() { + const N: usize = 200_000; + macro_rules! per_flush { + ($name:expr, $acc:ident) => {{ + let mut acc = $acc::new(); + let mut t: Vec = (0..15) + .map(|_| { + let t0 = Instant::now(); + for _ in 0..N { + black_box(&mut acc).flush(); + } + t0.elapsed().as_nanos() as f64 / N as f64 + }) + .collect(); + t.sort_by(f64::total_cmp); + black_box(acc.finish()); + let med = t[7]; + println!( + "{:<60} {med:8.3} ns/flush ({:.6} ns/64 amortized over {} blocks)", + $name, + med / B_FLUSH_BLOCKS as f64, + B_FLUSH_BLOCKS + ); + }}; + } + println!("-- E, flush overhead"); + per_flush!("B flush: 4 x I32x16 -> i64", AccB); + per_flush!("E3 flush: 3 x I32x16 -> i64", AccE3); + per_flush!("E5 flush: 2 x I32x16 + 2 x U64x8 reduce_sum -> i64", AccE5); } #[inline(never)] @@ -454,6 +495,37 @@ pub fn asm_e3_mac(acc: &mut AccE3, a: U8x64, negative: u64, low7: I8x64, top: u6 acc.mac(a, negative, low7, top) } #[inline(never)] +pub fn asm_e5_mac(acc: &mut AccE5, a: U8x64, negative: u64, low7: I8x64, top: u64) { + acc.mac(a, negative, low7, top) +} + +// Whole arithmetic loops, out of line, so the disassembly shows the steady +// state: accumulators in registers or spilled, and what runs per block. +#[inline(never)] +pub fn asm_b_loop(mags: &[U8x64], split: &[(u64, I8x64, I8x64)]) -> i64 { + let mut acc = AccB::new(); + for (a, (ng, l, t)) in mags.iter().zip(split) { + acc.mac(*a, *ng, *l, *t); + } + acc.finish() +} +#[inline(never)] +pub fn asm_e3_loop(mags: &[U8x64], anc: &[(u64, I8x64, u64)]) -> i64 { + let mut acc = AccE3::new(); + for (a, (ng, l, t)) in mags.iter().zip(anc) { + acc.mac(*a, *ng, *l, *t); + } + acc.finish() +} +#[inline(never)] +pub fn asm_e5_loop(mags: &[U8x64], anc: &[(u64, I8x64, u64)]) -> i64 { + let mut acc = AccE5::new(); + for (a, (ng, l, t)) in mags.iter().zip(anc) { + acc.mac(*a, *ng, *l, *t); + } + acc.finish() +} +#[inline(never)] pub fn asm_b_mac(acc: &mut AccB, a: U8x64, negative: u64, low7: I8x64, top: I8x64) { acc.mac(a, negative, low7, top) } @@ -498,11 +570,15 @@ fn main() { asm_e3_mac(&mut e3, a, black_box(0x55), l, black_box(0xF0)); let mut b = AccB::new(); asm_b_mac(&mut b, a, black_box(0x55), l, l); + let mut e5 = AccE5::new(); + asm_e5_mac(&mut e5, a, black_box(0x55), l, black_box(0xF0)); + black_box(e5.finish()); black_box((e4.finish(), e3.finish(), b.finish())); } if std::env::args().any(|x| x == "e") { bench_e("L1-resident, repeated 4096-value dot", 64, 2001, 1); bench_e("streamed", 1 << 20, 5, 2); + flush_costs(); return; } { diff --git a/tests/u9_abacus.rs b/tests/u9_abacus.rs index 81ce2f99..131032ad 100644 --- a/tests/u9_abacus.rs +++ b/tests/u9_abacus.rs @@ -12,7 +12,7 @@ mod carry16; mod anchor; use abacus::{carry_save3, planes, Abacus, AccAbacus, AccCsa, AccPopcnt, QueryC}; -use anchor::{bit7, sign_bytes, split_anchor, AccE3, AccE4, AccESum, QueryE}; +use anchor::{bit7, sign_bytes, split_anchor, AccE3, AccE4, AccE5, AccESum, QueryE}; use carry16::{carry_after, carry_before, AccD, QueryD, D_FLUSH_BLOCKS}; use kernels::{check, oracle, AccB, Block, NeverTouched, B_FLUSH_BLOCKS}; use ndarray::simd::{U16x32, U8x64}; @@ -401,3 +401,106 @@ fn e_worst_case_runs_cross_the_flush_bound() { assert_eq!(run_e(&a, &b), [oracle(&a, &b).unwrap(); 6]); } } + +/// E5 (vertical masked byte sums, #390) end to end and as a scan (b as the +/// prepared query), next to B on the same data. +fn run_e5(a: &[Raw], b: &[Raw]) -> [i64; 3] { + let (mut e5, mut s5, mut bk) = (AccE5::new(), AccE5::new(), AccB::new()); + for (ra, rb) in a.iter().zip(b) { + let (x, y) = (block(ra), block(rb)); + check(x, y).unwrap(); + e5.step(x, y); + s5.scan_step(x, &QueryE::new(y)); + bk.step(x, y); + } + [e5.finish(), s5.finish(), bk.finish()] +} + +#[test] +fn e5_all_511_by_511_pairs_are_exact() { + let mut n = 0usize; + for va in -255..=255i32 { + for vb in -255..=255i32 { + let lane = n % 64; + let (ma, pa) = encode(va); + let (mb, pb) = encode(vb); + let (mut am, mut bm) = ([0u8; 64], [0u8; 64]); + am[lane] = ma; + bm[lane] = mb; + let ap = if pa { u64::MAX } else { !(1u64 << lane) }; + let bp = if pb { u64::MAX } else { !(1u64 << lane) }; + let want = i64::from(va) * i64::from(vb); + assert_eq!(run_e5(&[(am, ap)], &[(bm, bp)]), [want; 3], "{va} x {vb}"); + n += 1; + } + } + assert_eq!(n, 511 * 511); +} + +#[test] +fn e5_full_blocks_with_mixed_signs_match_the_oracle() { + let mut s = 0x2545_F491_4F6C_DD1Du64; + let mut val = move || { + s = s + .wrapping_mul(6_364_136_223_846_793_005) + .wrapping_add(1_442_695_040_888_963_407); + ((s >> 33) % 511) as i32 - 255 + }; + for blocks in [1usize, 2, 7, 64, 300] { + let mut mk = || { + let mut m = [0u8; 64]; + let mut p = 0u64; + for (i, slot) in m.iter_mut().enumerate() { + let (mi, pi) = encode(val()); + *slot = mi; + p |= u64::from(pi) << i; + } + (m, p) + }; + let a: Vec<_> = (0..blocks).map(|_| mk()).collect(); + let b: Vec<_> = (0..blocks).map(|_| mk()).collect(); + // Mixed signs and both bit-7 states present, or the case proves little. + assert!(a + .iter() + .chain(&b) + .any(|(m, p)| *p != 0 && *p != u64::MAX && m.iter().any(|&x| x >= 128))); + assert_eq!(run_e5(&a, &b), [oracle(&a, &b).unwrap(); 3], "{blocks} blocks"); + } +} + +/// All-255 runs past three flush bounds, every sign mix; totals beyond `i32`. +#[test] +fn e5_worst_case_runs_cross_the_flush_bound() { + let n = 3 * B_FLUSH_BLOCKS + 5; + for (pa, pb) in [(u64::MAX, u64::MAX), (u64::MAX, 0), (0, 0), (0xF0F0_F0F0_0F0F_0F0F, 0x3333_CCCC_5555_AAAA)] { + let a = vec![([255u8; 64], pa); n]; + let b = vec![([255u8; 64], pb); n]; + assert_eq!(run_e5(&a, &b), [oracle(&a, &b).unwrap(); 3]); + } +} + +/// The anchor accumulators are reduced once per flush, never per block: +/// `n` blocks give `n / B_FLUSH_BLOCKS` flushes, plus one at `finish`. +#[test] +fn e5_reduces_the_anchor_only_at_flush() { + let n = 2 * B_FLUSH_BLOCKS + 7; + let (x, y) = (block(&([200u8; 64], 0x0F0F)), block(&([131u8; 64], u64::MAX))); + let mut e5 = AccE5::new(); + for _ in 0..n { + e5.step(x, y); + } + assert_eq!(e5.reductions, n / B_FLUSH_BLOCKS); +} + +/// Touched zero scores 0 in E5; never-touched is refused before E5 sees it. +#[test] +fn e5_touched_zero_scores_zero_and_never_touched_is_refused() { + let zero = ([0u8; 64], u64::MAX); + let big = ([255u8; 64], 0xAAAA_5555_AAAA_5555); + assert_eq!(run_e5(&[zero, big], &[big, zero]), [0; 3]); + let mut m = [200u8; 64]; + m[9] = 0; + let sentinel = block(&(m, !(1u64 << 9))); + assert_eq!(check(sentinel, block(&big)), Err(NeverTouched(1 << 9))); + assert_eq!(check(block(&big), sentinel), Err(NeverTouched(1 << 9))); +}