feat!: generate compact parsers with corpus profiles

[exKAZUu]

Why

Adopt the common profiled ABI 16 generator to reduce parsing time and compiled size. Profiles must be regenerated with the grammar and release metadata, so checked-in profile files would become stale.

Requirements

• required: Adopt the latest parser optimization mechanism in every WillBooster grammar repository, including TypeScript and TSX.

• required: Make all organization grammars faster and smaller with a common generation policy; repair tree-sitter defects or regressions discovered during adoption.

• required: Run complete-pr, review-booster, complete-pr, and release for every PR, in that order.

• required: Preserve existing syntax trees, queries, Unicode behavior, error recovery, and incremental parsing behavior.

• chosen: Generate profiled ABI 16 parsers and require WillBooster runtime 1.3.0 or later; publish a major grammar version because earlier runtimes cannot load ABI 16.

• chosen: Pin development Web and CLI runtimes, plus Rust where bindings are present, together at the release that supplies the profile command, so every validation and build uses the same generator and runtime.

• chosen: Record applicable corpus cases and Git-tracked example files with the same recorder defaults in every repository. Corpus-only Bash recording did not reduce native size; adding the maintained examples does, without per-grammar tuning.

• chosen: Regenerate the unprofiled baseline, record a fresh profile, and generate the profiled parser whenever generating or building Wasm, including releases. Profiles are ignored temporary files because grammar changes, generator changes, and release metadata invalidate their source fingerprint.

• chosen: Generate each distinct grammar path from tree-sitter.json once, treating an omitted path as the repository root, so aliases sharing a grammar do not retrain or regenerate it.

• chosen: Retain the current grammar definitions and parent grammar dependency versions because adopting the generator and runtime does not require changing the accepted syntax or tree structure.

• chosen: Use a common maximum of 128 dense parse states for recorded profiles. Across all nine organization variants this reduces both fresh/incremental time and compiled size with matched native O2/O3 and Wasm Os builds; Web also improves. The 256-state cap made Bash O3 output 160 bytes larger, so no per-grammar exceptions or build flags are needed.

• chosen: Use the already pinned Node runtime to read generation configuration and grammar names. The documented toolchain supplies Node but not jq, so adding jq would create an unnecessary mandatory build dependency.

• chosen: Use tree-sitter.json as the parser metadata input and set release versions only in configuration and package manifests. The only caller is build-release, which immediately regenerates parser.c, so directly patching generated metadata is redundant and unnecessarily requires an existing metadata block.

• chosen: Document grammar files, tree-sitter.json, applicable corpus cases, and Git-tracked examples as generation inputs, and require regeneration and committed output after input changes. Added or removed examples must be staged before generation because git ls-files supplies the training file list and still includes unstaged deleted paths. Skipping missing files would silently change the reviewed profile input set.

• chosen: Describe runtime 1.3.0 as the supported consumer minimum separately from the pinned development runtime. Cargo.lock and script/runtime-version select the runtime used for repository tests and fuzzing.

• chosen: Include configuration, corpus files/directories, and Git-tracked example timestamps in the existing Wasm freshness check, so local performance tests reject builds made from older profile inputs; CI also verifies regenerated source against the committed output.

• chosen: Materialize the Git-tracked example list before reading it, so a Git failure aborts generation instead of silently producing a corpus-only profile. A successful empty list remains valid for Kotlin, which has no tracked training examples.

• chosen: Compare the current tracked-example list with the list published only after successful generation as well as checking timestamps, so staging an old file or removing a tracked example cannot silently reuse a stale Wasm build.

• chosen: Include the generated parser header in Wasm source freshness checks because ABI 16 changes its language layout and a changed header must not silently reuse the earlier binary.

• chosen: Missing listed inputs fail with rebuild guidance; silently dropping an absent tracked example would train on different data instead of detecting the stale build.

• chosen: Use a per-run pending example-list file so overlapping successful generations do not consume each other's success marker; only completed generation publishes examples.z.

Customer Summary

Generation, Wasm builds, and release builds now record applicable corpus cases and tracked examples, then generate the compact parser. This is a breaking grammar release: consumers need WillBooster runtime 1.3.0 or later.

Technical Summary

script/generate drives the same unprofiled-generation, recording, and profiled-generation sequence for each distinct configured grammar path. build-wasm, build/ci, and release builds use it; temporary profiles remain under .tmp. Development Web/CLI runtime versions are pinned at 1.4.0, together with Rust where bindings are present. Grammar definitions and parent grammar versions are unchanged.

Testing

Measurements run on Darwin arm64 with Apple Clang 21.0.0 and Node 24.21.0; native O2/O3 are matched CLI builds. Compared fresh ABI 15 generation with profiled ABI 16 output using identical CLI compiler settings: native O2 and O3, Wasm Os. The same recorder default caps dense states at 128 across the organization. Negative percentages mean lower time or size. Times are geometric means of six input-file median ratios across six alternating native/Wasmtime rounds and five Web rounds, covering code, corpus, malformed, and Unicode inputs. These are corpus-level measurements, not guarantees for every possible input.

Grammar	Backend	Fresh time	Incremental time	Binary size
javascript	Native O2	-10.86%	-12.62%	-21.17%
javascript	Native O3	-11.11%	-12.95%	-21.17%
javascript	Wasmtime Os	-8.02%	-10.29%	-21.78%
javascript	Web runtime 1.3.0	-12.78%	-13.99%	Same Wasm above

Full-node hashes, error states, and fresh/incremental trees agree across layouts and native/Wasmtime/Web backends. The final 128-state generation passes all 199 applicable corpus cases both natively and through Wasmtime, and local bun run verify passed. Actual generation scripts reproduce the measured sources. Rust bindings also pass actual consumer checks on the published minimum runtime 1.3.0 (ABI, embedded metadata, complete trees, errors, edits, and configured queries). Released CLI and dependency validation remain pending. Full suites run in PR CI; release verification checks the actual published npm Wasm and Rust artifacts against the merged commit.

The release script also passes in an isolated working copy with planned major-version metadata, using the source-built CLI and published runtime 1.3.0. Its generated layout matches the measured source except for metadata; resulting Wasm metadata, complete trees, edits, and configured queries pass. This is a local release-build probe, not a published-artifact check.

The actual local npm tarball and Rust crate (when present) pass the same consumer checks on runtime 1.3.0, including shipped queries, after packaging. Actual published artifacts and provenance remain pending.

The published runtime 1.4.0 Web JS/Wasm and native C/header files are byte-identical to runtime 1.3.0 used for the measurements. Actual runtime 1.4.0 package provenance, grammar loading/tree/edit/query checks, and legacy ABI compatibility also pass.

Existing corpus, incremental parsing, and Wasm unit test files pass with runtime/CLI 1.4.0.

The current pushed head passes all PR checks and is mergeable (check-pr-ready: PASS).