Skip to content

Publish training-free choice-token readout matrix - #22

Merged
weitianxin merged 11 commits into
mainfrom
feature/choice-readout-evals
Oct 2, 2026
Merged

weitianxin merged 11 commits into
mainfrom
feature/choice-readout-evals

Conversation

@weitianxin

Copy link
Copy Markdown
Collaborator

Summary

  • expose training-free choice-token readout as --readout choice, with legacy aliases retained
  • evaluate frozen 4B/27B, Pointer 4B/27B, and Direct-Token 4B on one matched matrix
  • select stack weight and temperature only on Transfer-v9 development, then freeze them for held-out Transfer, Typed Decisions, and JevJudge text
  • publish validated JSON, a README SVG, complete method tables, and a deterministic 23-page merged technical report
  • support all 52 one-token option IDs and memory-linear attention for long JevJudge text records

Headline results

  • Direct-Token 4B tuned stack: 79.83% Transfer test, 67.65% Typed, 59.25% JevJudge text
  • Pointer 27B tuned stack: 89.10% Transfer test, 73.30% Typed, 90.04% JevBench; native remains best on JevJudge text at 66.44%
  • native parity gates reproduce the canonical README rows before held-out evaluation

Verification

  • 97 relevant API/readout/evaluator/builder/playground tests passed; external-server tests require a running endpoint
  • 5 report producer/consumer, in-place rebuild, page-preservation, and byte-determinism tests passed
  • package wheel and sdist build succeeded
  • result JSON/SVG and the 23-page PDF rebuild byte-identically
  • full-suite run reached 455 passed; remaining failures are from the cluster eval environment loading CUDA-only causal-conv kernels in CPU tests or requiring an external HTTP server

@weitianxin
weitianxin merged commit 3f6da42 into main Oct 2, 2026
5 checks passed
@weitianxin
weitianxin deleted the feature/choice-readout-evals branch October 2, 2026 02:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant