Repository navigation
An invalid pattern in a JSON schema throws and kills the whole run #8
Description
Desired outcome
An invalid pattern in a user's JSON schema produces a scorer failure, not a crash.
Why it matters
validate() in src/scorers/json-schema.ts compiles a user-supplied regex with no guard:
if (schema.pattern && !new RegExp(schema.pattern).test(value))
errors.push(`${path}: does not match pattern ${schema.pattern}`);schema comes straight from the eval file, so pattern is arbitrary user input. A typo such as "[a-z" makes new RegExp throw a SyntaxError, which propagates out of validate, out of the scorer, and up through the runner, taking the whole eval run with it. The user sees a raw stack trace mentioning RegExp and nothing that points at which case or which schema key is at fault.
The sibling scorer already handles this properly. src/scorers/regex.ts wraps compilation in try/catch and returns:
reason: `invalid regex: ${(err as Error).message}`,so the same class of mistake is a clean, attributable failure in one scorer and a crash in the other.
The regex is also recompiled on every call, once per case per run, which is wasteful but secondary.
Steps
- Wrap the
new RegExp(schema.pattern)intry/catchinsidevalidate(). - On a compile error, push a descriptive entry into
errors, for example`${path}: invalid pattern ${schema.pattern}: ${message}`, so the failure carries the JSON path and is reported like any other validation error. - Add a test in
tests/scorers.test.tswith a schema containing an unclosed character class, asserting the scorer returns a failing result rather than throwing.
Small and self-contained, one function.
Claiming this
Comment below to claim it. A reply usually comes within a day.
Activity
vsolano9 commentedon Aug 23, 2026
I'd like to work on this reopened issue. I'll keep the fix inside the JSON Schema validator, add regression coverage proving an invalid nested pattern returns a path-attributed scorer failure instead of throwing, and run the full repository gate.
github-actions commentedon Aug 23, 2026
It's yours, @vsolano9. Marked this as
claimed.Take your time, and shout in this thread if you get stuck. If there's no activity for 14 days the claim is released so someone else can pick it up.
github-actions commentedon Sep 5, 2026
The linked pull request #22 was closed without being merged, so this issue is open again.
The work is still needed. Anyone is welcome to pick it up, just comment here and it's yours.
github-actions commentedon Sep 5, 2026
The linked pull request #22 was closed without being merged, so this issue is open again.
The work is still needed. Anyone is welcome to pick it up, just comment here and it's yours.
github-actions commentedon Sep 5, 2026
The linked pull request #22 was closed without being merged, so this issue is open again.
The work is still needed. Anyone is welcome to pick it up, just comment here and it's yours.
github-actions commentedon Sep 5, 2026
The linked pull request #22 was closed without being merged, so this issue is open again.
The work is still needed. Anyone is welcome to pick it up, just comment here and it's yours.