Skip to content

SCRUM-1078-maintain verify: report joins whose keys have no overlap - #473

Open
emnarahmouni-exmergo wants to merge 3 commits into
mainfrom
SCRUM-1078-issue-228
Open

emnarahmouni-exmergo wants to merge 3 commits into
mainfrom
SCRUM-1078-issue-228

Conversation

@emnarahmouni-exmergo

@emnarahmouni-exmergo emnarahmouni-exmergo commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • maintain verify reports a selected model's own equality joins whose keys have no (or too little) value overlap, reusing the exact overlap probe explore relationships --verify already runs (explore.relationships.verify_relationships) rather than authoring a second one.
  • A join at or near a complete orphan rate is join_zero_overlap (high severity, the most damaging and least visible defect this sweep can report); one with real but non-catastrophic orphans is join_orphans (medium) — also where a left join whose right side is legitimately sparse lands, reported at its measured fraction rather than characterized as broken.
  • JOIN ... USING (col) is recognized as the equality it implies, the same as a written ON, but only turned into a candidate when exactly one relation is already in scope to pair it against — a chained USING where the column was introduced by an earlier join (not the original FROM) is skipped rather than guessed against the wrong table.
  • A join's condition must be a plain conjunction of column-to-column equalities to become a candidate: an OR, a range predicate, or any other non-equality conjunct causes the whole join to be skipped (never probed on a partial subset of its real condition, which can under- or over-report the true orphan rate), and the skip is named in warnings rather than silent.
  • A composite equality (ON a.x = b.x AND a.y = b.y) is measured as one combined key, not two independent single-column probes. A self-join is only excluded when both sides are the literal same key on the same relation (t.col = t.col); one on different columns (employees.manager_id = managers.id) is a real, checkable join.
  • A join's key is resolved through as many CTE or inline-subquery hops as it takes to the physical relation and column it actually reads — including a column renamed by a CTE, computed, projected from a joined relation rather than the CTE's own FROM, or passed through SELECT * — refusing (and skipping the whole join) rather than guessing wherever that isn't safely determinable: a computed projection, an ambiguous star, a qualified table reference sharing a CTE's short name, a scope that filters or aggregates before the join reads it (which would measure the probe against the wrong row set), or a column read directly off a set-operation CTE's output (which has no single physical source across its branches).
  • Every scope the query reaches is visited, not only the driving-parent chain: a join inside a CTE that's itself joined in later, a join inside an inline subquery, and every branch of a top-level or nested UNION/INTERSECT/EXCEPT are all checked independently.
  • join_contract_plan parses each model's compiled SQL in the connector's actual dialect (matching row_population_plan's existing convention), not hardcoded to DuckDB.
  • The overlap probe is a real warehouse scan, priced and offered the same way row population's row counts already are. Since VerifyResult.pending_offer holds only one offer and both row population and join contract can each need a scan on a metered connector, the two share one combined handshake naming both axes, so confirming buys exactly what the estimate described instead of a metered connector silently seeing only whichever axis asked last.
  • New shared primitives in sql_shape.py: equality_column_pairs (raw columns behind equality_columns's existing rendered-text pairs) and conjunctive_equality_pairs (also respects the condition's Boolean structure).

Test plan

  • Every reviewer-reported finding reproduced directly (a minimal SQL snippet through _model_joins) before touching code, and re-verified against the same snippet after the fix
  • 40+ new tests covering: CTE column-alias/rename resolution, columns projected from a joined (not FROM) relation, OR predicates, range predicates, composite keys, JOIN ... USING (simple and ambiguous-chained), self-joins (real and trivial), joins inside joined CTEs, joins inside inline subqueries, qualified tables sharing a CTE's short name, filtered/aggregated CTEs, set operations at the top level/inside a CTE/inside a subquery, the combined-handshake pricing and caching, and the connector-dialect parse
  • tests/maintain/test_verify.py — 118/118 passed
  • Full regression sweep: tests/maintain/ tests/transform/ tests/explore/ tests/test_sql_shape.py — clean through the prior four rounds (1998 passed/30 skipped pre-existing/0 failed most recently); sweep after the dialect fix is finishing now, will confirm before merge
  • ruff check / ruff format clean on all touched files
  • Zero em-dashes in code, tests, and CHANGELOG

Closes : #228

The model's own compiled SQL was already a real join-contract candidate
(customers inner-joined to events), which the pinned-DuckDB parse bug
this branch already fixed had been silently masking: every prior sweep
passed only because the join was never detected, not because the fixture
correctly answered a probe.

Update the offer axes assertion to name both axes now that both have
real work, and teach the fake BigQuery client's row_resolver to answer
the overlap-probe query shape alongside the row-count one it already
knew, reporting the join healthy so it adds no finding these tests are
not about.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
sqlglot now parses from_json() into a dedicated FromJson class instead
of falling through to Anonymous, which the name-based allowlist no
longer catches. Uses the same getattr/filter pattern _QUERY_ROOTS
already relies on, so it's a no-op on the pinned version.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

maintain verify: report joins whose keys have no value overlap

1 participant