Skip to content

Add optional property-based testing suite (src/test/prop) - #51

Open
gburd wants to merge 338 commits into
masterfrom
hegel
Open

gburd wants to merge 338 commits into
masterfrom
hegel

Conversation

@gburd

@gburd gburd commented Oct 13, 2026

Copy link
Copy Markdown
Owner

Summary

An optional, opt-in property-based testing (PBT) layer for PostgreSQL built on the Hegel engine (libhegel), living entirely under src/test/prop. Off by default, gated on three conditions (explicit configure flag --with-hegel / -Dprop_tests=enabled, the engine being present, and TAP tests enabled); if any piece is missing under the opt-in flag it is a hard configure error, and with the flag off nothing changes.

This is a large series (one commit per area) opened primarily so CI runs the suite. Nothing in the server is modified; this only adds a test tree plus the build plumbing to wire it in.

What's here

  • Harness (pg_prop.c/.h, pg_prop_sql.c): a small plain-C wrapper over libhegel's C ABI, a PG_PROP() registry, TAP output, a fork-based isolate() for code that may crash, and libpq oracle helpers. CLI: pg_prop <area> [-n iters] [-s seed] [property ...].
  • Two tiers: Tier A runs in-process (t/001_prop.pl, 12 areas: path, jsonapi, encoding, unicode_case/norm, pglz, base64, prng, crypto, scram, binaryheap, bitutils). Tier B drives SQL against a live node via libpq (t/002_sql_props.pl, ~246 areas).
  • ~250 scalar/type/aggregate/window/planner/executor/MVCC/contrib areas, each a differential or round-trip oracle (two ways that must agree), one commit per area.
  • Query-level areas (src/test/prop/sql/): a shared random-table + predicate generator feeding metamorphic oracles that test the planner and executor rather than individual functions:
    • query_tlp, query_tlp_agg, query_tlp_distinct -- Ternary Logic Partitioning (rows, aggregates, DISTINCT/set-ops)
    • query_norec -- non-optimizing reference, including across a join
    • query_cardinality -- result-size monotonicity laws
    • query_pqs -- Pivoted Query Synthesis (single query vs single known-true fact)
    • query_planadvice -- the same query run free vs steered via contrib/pg_plan_advice (scan/join method, join order, merge join, spill); result must not depend on the chosen plan
  • Extension points so an extension can (a) add its own area, or (b) augment the SQL query generator with its own type via three hooks (query_gen.setup/column/predicate), after which every query-level oracle covers that type with no new oracle code. sql/prop_query_ext_citext.c is a complete worked example.

Validation

Full TAP suite green on a cassert build: 12 Tier-A + 246 Tier-B subtests, zero warnings, run under the TAP driver's own seed stream. On a branch-coverage build the query-level areas drive real planner/executor coverage (e.g. clausesel.c ~72% branches; a single plan-advice property took nodeMergejoin.c from 0% to ~30% branches).

See src/test/prop/README and src/test/prop/DESIGN for the full rationale, the when-to-use/when-not guidance, the measured coverage story, and the extension mechanism.

No server bug is fixed or reported here; findings surfaced during development are property/oracle-soundness issues recorded out of tree.

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant