Skip to content

test: add never-panic harness for lexer and parser - #93

Open
shayyz-code wants to merge 2 commits into
mainfrom
test/80-never-panic-harness
Open

shayyz-code wants to merge 2 commits into
mainfrom
test/80-never-panic-harness

Conversation

@shayyz-code

@shayyz-code shayyz-code commented Oct 14, 2026 •

Copy link
Copy Markdown
Owner

Summary

Adds tests/never_panics.rs, a zero-dependency harness that checks the lexer and parser return errors instead of panicking. It drives Parser::parse_checked, which runs the lexer. The lexer's own checked API is crate-private, and execution is out of scope until the interpreter has budgets.

Inputs:

  • every char-boundary prefix of examples/*.poo and tests/conformance/**/*.poo (the conformance directory is optional, so this works before test: add conformance corpus harness #79 lands)
  • random insert/delete/replace/duplicate edits of those files, using an alphabet with quotes, slashes, brackets, digits, <:, >>, .. and non-ASCII text (မ, é, emoji, zero-width space, BOM)
  • random sequences of v0.1 tokens and statement fragments; half of them are wrapped in a typed function body so they reach the parse-time type checks
  • random Unicode scalar values (no surrogates)
  • pathological inputs, one generator each: 10k-deep ( / [ / { / if / not / - nesting, a 10k-term binary chain, 1 MiB of whitespace, 1 MiB of // // comments, a 1 MiB comment body, 30-digit integer, range and float literals, a 100k-char identifier, a 1 MiB string, unterminated string/escape/char/comment at EOF, non-ASCII identifiers and strings, NUL and control characters, a BOM prefix, and empty input

How it works:

  • A deterministic xorshift64* PRNG is seeded from POO_FUZZ_SEED (decimal or 0x hex). POO_FUZZ_ITERS sets the number of random inputs per generator (default 5000). Each generator gets its own stream, so skipping one does not change the inputs of another.
  • Inputs run one at a time on a long-lived worker thread with an 8 MiB stack, each inside catch_unwind. A quiet panic hook records worker panics so they don't flood the output. Panics on other threads still print, and the previous hook is restored before reporting. If a parse doesn't finish within 30 s, the test fails as a hang instead of blocking.
  • Unknown panics are grouped by source location, up to 10 of them. Each one is shrunk by greedy chunk deletion plus prefix and suffix trimming, bounded to 2 s and 5000 attempts per failure. The report prints the escaped minimized input and its panic message.
  • tests/never_panics/known_crashers.txt lists known crashers with their issue. A generator entry skips that generator, which is needed for stack overflows because they abort the process. A panic entry turns a caught panic whose message contains the key into a "known" result.
  • POO_FUZZ_INCLUDE_KNOWN=1 also runs the skipped generators, and POO_FUZZ_ONLY=<generator> runs just one. Together they confirm a fix.

Issue

Closes #80

Verification

  • just check: fmt, check, clippy -D warnings, cargo test, mdBook build and test, dist generate --check, cargo deny check. All pass.
  • cargo test --test never_panics -- --nocapture on this machine (debug build): about 2 s for the whole run (1.6 to 2.0 s across runs). Summary:
    never_panics: seed=0xb00cafe5eed00080 iters=5000 corpus_files=2 inputs_tried=16273 known_skipped=9 known_panics=72 failures=0
  • Also passes with seeds 1 and 0xdeadbeef.
  • Exploratory runs with no known-crasher filtering: 100k mutations with two seeds, 200k token sequences, 40k Unicode strings, and each pathological generator in its own process.
  • Removing the #48 and #46 entries makes the test fail as expected, printing minimized inputs "9999999999999999999", "12345678901234567890.." and "poof f(a int s str)>>int{return x;}".

Known crashers found:

Input Effect Issue
Integer literal above i64::MAX, e.g. 9999999999999999999 or ….. (lexer.rs lines 170 and 184) panic #48
1 MiB of whitespace stack overflow #49
1 MiB of // // comments: the lexer recurses once per comment (same pattern as #49) stack overflow #95
10k-deep (, [, if true {, not, - (balanced or not): the parser overflows an 8 MiB debug stack at 1,000 to 2,000 levels stack overflow #94
Undefined variable: in parse-time type inference, e.g. poof f(a int)>>int{return x;} panic #46, #96
Undefined function: in parse-time type inference, e.g. poof g()>>void{return n();} panic #45, #96
Unsupported binary operation:, e.g. poof f(s str){return "">'';} panic #50, #96
Unexpected expression: (listed from the #47 audit report; the default seed does not hit it) panic #47, #96

#45, #46, #47 and #50 each cover only specific programs. #96 tracks making parse-time type inference return errors instead of panicking in general. The second commit, test: link known crashers to tracking issues, puts these issue numbers into known_crashers.txt.

No hangs were found, and no panics outside these families.

  • Each commit is a self-contained Conventional Commit and passes just check

Documentation and Visual Impact

None. This only adds a test, and user-visible behaviour does not change. Usage is documented in the module comment of tests/never_panics.rs and in the header of known_crashers.txt.

🤖 Generated with Claude Code

shayyz-code and others added 2 commits October 14, 2026 12:15
Feed Parser::parse_checked every prefix of the example and conformance
files, random edits of them, random token sequences, random Unicode, and
pathological inputs (deep nesting, 1 MiB of whitespace or comments,
30-digit literals, a 100k-char identifier, unterminated literals).

Each input runs under catch_unwind on a worker thread with an 8 MiB
stack. Unknown panics fail the test with a minimized input; known stack
overflows and panics are listed with their issue in
tests/never_panics/known_crashers.txt. A deterministic xorshift64* PRNG
is seeded from POO_FUZZ_SEED and sized by POO_FUZZ_ITERS. The default
run takes about 2 s in a debug build.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Point the deep-nesting overflows at #94 and the comment-run overflow at
#95, and cite the type inference umbrella issue #96 next to #45, #46,
#47 and #50, which each cover only specific programs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test: add never-panic harness for lexer and parser

1 participant