perf: speed up SQL auto-categorization

[nc-dirknilius]

Description

Motivation

SqlModel.is_breaking_change currently calls SQLGlot's general-purpose diff implementation to answer a much narrower question: whether a rendered query changed exclusively through added SELECT projections.

SQLGlot's tree diff computes candidate matchings across both ASTs, introducing quadratic work in its matching phases.
SQLMesh performs this work synchronously for each model during auto-categorization. We encountered the resulting
amplification in a project with more than 600 models: a broadly used macro change can require hundreds of models to
be categorized, repeatedly paying the superlinear cost for large rendered queries and causing sqlmesh plan to spend
more than an hour in categorization without completing; the plan was cancelled at that point.

The existing _additive_projection_change fallback handles cases where SQLGlot emits spurious Move or Update edits, such as repeated cast types. However, the fallback runs only after the expensive tree diff has completed, so it does not address the performance problem.

What changed

This replaces the general tree diff and its fallback with a specialized AST comparison:

• Walk both query ASTs in lockstep.
• Require identical node types, scalar arguments, and non-projection child lists.
• Treat each previous SELECT projection list as an ordered subsequence of the current list.
• Allow new projections before, between, or after existing projections.
• Preserve duplicate projections and structurally similar expressions such as repeated CAST types.
• Apply the same comparison recursively to CTEs, EXISTS queries, UNIONs, and other nested query structures.
• Return the conservative, undetermined result for every change that cannot be proven to be projection-only.

The comparison is linear in the traversed AST and projection lists rather than performing general candidate matching across the trees.

Safety and behavior

The specialized comparison preserves SQLMesh's conservative categorization rules:

• Removed, replaced, or reordered projections remain undetermined.
• Changes to FROM, WHERE, GROUP BY, ORDER BY, DISTINCT, or other query structure remain undetermined.
• Directly projected UDTFs remain undetermined because they can change row cardinality.
• A UDTF is accepted only when its nearest Subquery ancestor is contained within the added projection.
• Mid-list additions remain undetermined when GROUP BY or ORDER BY uses ordinal references, because inserting a projection can shift the referenced output.
• Changes inside an existing scalar-subquery projection remain undetermined.
• Any failed structural comparison returns None, retaining SQLMesh's conservative fallback behavior.

There are no public API or configuration changes.

Performance

Measured on main at e075200 using Python 3.13.13 and SQLGlot 30.8.0 on macOS ARM64.

Both ASTs were parsed before timing. Each benchmark copied the previous query and added one scalar projection. The measurements time only the existing sqlglot.diff call and the specialized comparison.

Input	AST nodes	sqlglot.diff	Specialized comparison	Speedup
Synthetic, 100 projections	1,104	87.9 ms	1.4 ms	62×
Synthetic, 500 projections	5,504	2.590 s	7.4 ms	351×
Synthetic, 1,000 projections	11,004	11.126 s	14.0 ms	797×
Real-world Redshift model	16,157	12.442 s	20.6 ms	605×

These are single-process wall-clock measurements intended to show the order-of-magnitude difference; parsing and rendering are excluded from both sides.

A representative breaking change on the same large model retained the conservative result while dropping comparison time from approximately 12.1 seconds to 20 milliseconds.

Test Plan

Added coverage for:

• Existing projection additions, removals, replacements, and reordering.
• Repeated CAST types.
• A new projection identical to an existing projection.
• Direct and aliased UDTFs.
• UDTFs inside projection subqueries.
• UDTFs inside derived-table projections.
• CTE projection additions.
• Projection additions inside EXISTS.
• Changes inside existing scalar-subquery projections.
• Projection additions across UNION branches.
• Nested ordinal ORDER BY safeguards.

Validation performed:

• make style
  • Passed: ruff, ruff-format, mypy, and migration validation.
• python -m pytest tests/core/test_snapshot.py -k 'categorize_change_sql' -q
  • 8 passed.
• Differential comparison of 500 generated projection and non-projection mutations against the current implementation.
  • No categorization mismatches.
• make fast-test
  • The exact target was run twice. Its parallel phase consistently completed 2,576 tests successfully, skipped 4,
    and reported one order-dependent error in
    tests/dbt/cli/test_selectors.py::test_select_by_dbt_names[dbt_select6-expected6].
  • The error occurs when another DBT test's temporary namespace package named macros remains in the same xdist
    worker and dbt-common attempts to inspect it. No changed code is present in the traceback.
  • Running all 14 test_select_by_dbt_names cases serially passes.
  • Running the parallel fast suite without that parametrized test passes: 2,563 passed and 4 skipped.
  • The phases that follow the parallel phase in make fast-test also pass when invoked separately:
    3 isolated, 1 registry_isolation, and 159 dialect_isolated tests.

Checklist

• I have run make style and fixed any issues.
• I have added tests for my changes.
• All existing tests pass (make fast-test).
  • All tests pass when the order-dependent DBT selector case is separated from the parallel run, as detailed above.
• My commits are signed off (git commit -s) per the DCO.

[mday-io]

Nice find and fix. Confirmed significant speedups myself (160x on a 400 projection model). Looks like all of the categorizations are safe and / or safer (in the case of ORDER BY 2 being NON_BREAKING and now classified as undetermined).

Only one additional easy win - the ORDER BY 2 fix won't kick in when ORDER BY sits after a UNION. Since you've already modified that area, worth a patch.

Some other nits listed in-line

[mday-io]

nit: these two lists could probably just be a boolean and a last-match index — the only question being answered at the end is "did anything get added before the last matched projection?" The appends in the trailing loop can never trip that check anyway (they're all past the last match), so they're effectively dead work. Optional — the any(...) at the bottom just takes a moment to unpack as written.

(tiny thing: list[int] here vs t.List everywhere else in the file)

[mday-io]

nit: the name is a little misleading — it returns True when the lists don't match, they just differ by safe additions. Maybe something like _projections_only_safely_added? Similar thought on _is_valid_added_projection — "valid" undersells what it checks, which is really "won't change row counts".

[mday-io]

As mentioned:

One case slips past this guard: when the ORDER BY 2 comes after a UNION, the ordinal lives on the Union node rather than the Selects — so adding a column mid-list in both branches still comes out NON_BREAKING even though the "2" now points at a different column. The old code had the same blind spot, so it's not a regression, but since this rewrite makes it easy to reach: checking ordinal refs on the nearest set-op ancestor when there's a mid-list add would close it. A UNION + ORDER BY 2 case in the new parametrized test would pin it down.

[mday-io]

nit: one more case that would be nice here: a plain SELECT a, c ... ORDER BY 2 → SELECT a, b, c ... ORDER BY 2. That one actually changed behavior in this PR (used to be NON_BREAKING, now undetermined — which seems like the right call), so pinning it down would be good.