Skip to content

Handle every error kind the test schema's invalid attribute defines - #131

Open
bryantaustin13 wants to merge 1 commit into
mainfrom
invalid-attribute-error-kinds
Open

bryantaustin13 wants to merge 1 commit into
mainfrom
invalid-attribute-error-kinds

Conversation

@bryantaustin13

Copy link
Copy Markdown
Contributor

testSchema.xsd defines five values for InvalidType — false, syntax, semantic, execution and true — four of which say the test expects the engine to fail, differing only in how. The runner recognised only true and semantic, so a test declaring invalid="syntax" fell through to value comparison and was scored a failure even though the engine had produced exactly the syntax error the test asked for.

The semantics were duplicated across five call sites, so they now live in one place as two predicates:

  • expectsError() — all four error kinds. The test passes only if the engine errored, and needs no to compare against.
  • preventsTranslation() — syntax and semantic only. Those cannot survive CQL-to-ELM translation, so build-cql must leave them out of a generated library; one such define would fail the translation of every define around it. Runtime and execution errors translate fine and are kept.

Two further sites needed the same widening and would otherwise have broken:

  • The results schema enumerated only ["true", "false", "semantic"], so writing invalid="syntax" into a report would have failed results validation.
  • build-cql excluded only semantic from generated libraries, so a syntax-invalid define would have been emitted and broken its whole library. Both currently affected tests are exactly that case.

Verified against the reference engine: 1639/170 -> 1641/168, with the two invalid="syntax" tests (CeilingIntegerGreaterThanMaxInteger and CeilingIntegerLessThanMinInteger) passing and no other test changing status.

testSchema.xsd defines five values for InvalidType — false, syntax, semantic,
execution and true — four of which say the test expects the engine to fail,
differing only in how. The runner recognised only `true` and `semantic`, so a
test declaring `invalid="syntax"` fell through to value comparison and was
scored a failure even though the engine had produced exactly the syntax error
the test asked for.

The semantics were duplicated across five call sites, so they now live in one
place as two predicates:

- expectsError() — all four error kinds. The test passes only if the engine
  errored, and needs no <output> to compare against.
- preventsTranslation() — syntax and semantic only. Those cannot survive
  CQL-to-ELM translation, so build-cql must leave them out of a generated
  library; one such define would fail the translation of every define around
  it. Runtime and execution errors translate fine and are kept.

Two further sites needed the same widening and would otherwise have broken:

- The results schema enumerated only ["true", "false", "semantic"], so writing
  invalid="syntax" into a report would have failed results validation.
- build-cql excluded only `semantic` from generated libraries, so a
  syntax-invalid define would have been emitted and broken its whole library.
  Both currently affected tests are exactly that case.

Verified against the reference engine: 1639/170 -> 1641/168, with the two
invalid="syntax" tests (CeilingIntegerGreaterThanMaxInteger and
CeilingIntegerLessThanMinInteger) passing and no other test changing status.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant