Skip to content

feat: a scan reads each database as of one instant, and fixes from a version and platform matrix - #149

Merged
jasdeepkhalsa merged 12 commits into
mainfrom
feat/consistent-reads-and-matrix-fixes
Oct 18, 2026
Merged

jasdeepkhalsa merged 12 commits into
mainfrom
feat/consistent-reads-and-matrix-fixes

Conversation

@jasdeepkhalsa

@jasdeepkhalsa jasdeepkhalsa commented Oct 18, 2026 •

Copy link
Copy Markdown
Member

One commit per change, each with its own tests. Most came out of a smoke matrix: every pair of Supabase 15 and 17 and PostgreSQL 14–18, each way. Each pair ran a first sync, drift with live rows on the target (renames, a retype under a generated column and views, an enum label, --prove), and a snapshot and restore.

Commit What it does
A revoke on an object the same apply drops is done A view dropped by the schema fix took its grants with it. When the drop ran first, the roles check's revokes on it failed with does not exist and the run exited 1. Such a revoke now counts as applied, with a note. One whose object a later fix creates is still retried. Fixes #147.
Warn before a restore or apply needs more locks than the target has The statements are counted (relations, indexes, TOAST and identity sequences) against the target's lock table, less what other sessions hold. When they look unlikely to fit, the run says so before starting, and names --no-transaction and max_locks_per_transaction as the ways round it.
Compare grants on schemas USAGE on a schema of the project's own wasn't compared, so a target without it read as matching while its tables were out of the Data API's reach. PostgreSQL 15's change to the default CREATE on public isn't compared across that version boundary.
A snapshot keeps grants on schemas and default privileges roles.sql held table, column and routine grants. Restore lost schema USAGE and every ALTER DEFAULT PRIVILEGES.
Leave out a fix an older target cannot parse From a newer source, UNIQUE NULLS NOT DISTINCT on 14 or a VIRTUAL column on 17 failed with a syntax error and rolled back the whole apply. On an older target that fix is now left out and reported, and the rest applies. --prove rehearses it the same way.
Leave Supabase's own default privileges, and pgmq's tables, out of the schema check supabase_admin's default privileges are the platform's, and postgres can't change them. pgmq's tables are compared by the queues check.
Offer only the extension changes the target can make A version mismatch was always fixed with UPDATE TO the source's version, even downwards (pg_cron 1.6 against 1.6.4), which failed the apply. The target's update paths now decide. An extension in its own schema creates that schema first; plain PostgreSQL may lack extensions.
Restoring into an older server leaves out what it cannot hold The same as the apply, for restore: the snapshot's version is read from pg_dump's header.
A scan reads each database as of one instant The checks ran many concurrent queries, and a change made meanwhile was seen by some of them only. Each database is now read in one exported snapshot that every pooled connection joins, released before anything is applied. --no-snapshot reads query by query.
An extension in a system schema is created without making the schema pg_cron lives in pg_catalog on Supabase, and CREATE SCHEMA "pg_catalog" is refused.
An extension the target does not ship is left out, schema and all Once an extension's fix made its schema first, it no longer began with CREATE EXTENSION, and one the server doesn't ship (pg_graphql on plain PostgreSQL) failed the whole apply instead of being left out.
--prove matches a NOT NULL constraint by its column, not its name On PostgreSQL 18 a NOT NULL is a named constraint, and a renamed table keeps its old names. Compared with the source's structure made fresh, a rename from an older source read as constraints missing, and the migration was refused.

Tests:

  • New unit and e2e tests, including reads as of a held snapshot against a real server.
  • The unit suite and lint pass.
  • The release smoke battery passes.

🤖 Generated with Claude Code

jasdeepkhalsa and others added 10 commits October 18, 2026 11:55
A view the target has and the source lacks is dropped by the schema
check's fix and its grants revoked by the roles check's. When the drop
ran first, each revoke failed with "does not exist", was reported as
left out, and the run exited 1 — though the view, and its grants with
it, were gone as intended. A fix that only revokes privileges on one
object, still failing that way once retrying it after the others no
longer helps, is now counted as applied, with a note that there was
nothing left to revoke. A revoke whose object a later fix creates is
still retried after it.

Fixes #147

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… has

A restore or apply runs in one transaction, which holds a lock on every
relation it creates, alters or drops until it commits, in a table the
whole server shares. A large one ran out part way, after minutes of
work, and rolled everything back with "out of shared memory".

The statements are now counted before the transaction starts — each
relation once, and what a new table brings with it: its TOAST table and
index, an index per key, an identity sequence — against the target's
lock table, sized from max_locks_per_transaction and its backend slots,
less what other sessions hold. When they look likely not to fit, the
run says so first, with both numbers, raising max_locks_per_transaction
and --no-transaction as the ways round it. It warns rather than
refuses: the table can grow into spare shared memory, so a count over
its size may still fit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
USAGE on a schema of the project's own is what lets a role reach the
tables in it at all, and it was not compared: a target without it read
as matching, and a sync left the schema out of the Data API's reach.
The roles check now reads schema grants as it reads table and routine
grants, offers the GRANT or REVOKE, and leaves the grants on a schema
only the target has to the schema check, as it does for routines.

PostgreSQL 15 stopped granting CREATE on public to everyone, so between
servers either side of that change the difference is each version's
default; copying it into a newer server would undo its hardening, and
it is not compared there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
roles.sql held the grants on tables, columns and routines, and the
schema layer is dumped without privileges, so a restore lost USAGE on
the project's own schemas — leaving their tables out of reach — and
every ALTER DEFAULT PRIVILEGES, so new tables were no longer granted
what they were on the source. Both are captured now: a schema's grants
as GRANT ... ON SCHEMA, and each role's defaults as the ALTER DEFAULT
PRIVILEGES that make them, against PostgreSQL's built-in defaults for a
global entry. The roles they name are created first on a target that
lacks them, as for any grant.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A source newer than the target writes what the target has no syntax or
feature for: UNIQUE NULLS NOT DISTINCT before PostgreSQL 15, a VIRTUAL
generated column before 18, security_invoker before 15. The fix failed
with a syntax error, and the whole apply rolled back with it.

The apply now knows the source's version. On a target that is older, a
syntax error, an unsupported feature or an unknown parameter means the
target cannot hold that fix: it is left out and reported, as one
needing a missing extension is, and the rest is applied. --prove
rehearses it the same way. On a target as new as the source those are
still real failures and roll everything back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…of the schema check

dbdiff now compares default privileges, and between plain PostgreSQL
and Supabase it offered to change those supabase_admin holds — the
platform's, which postgres cannot alter — so the apply failed. Default
privileges for a platform role are left out, as their grants are;
postgres's own are the project's and still compared.

pgmq's schema holds the queues' tables, which the queues check compares
through pgmq; the schema check no longer reads them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A version mismatch was fixed with ALTER EXTENSION ... UPDATE TO the
source's version whichever way it went. PostgreSQL has no way down a
version, and a server may not ship the one asked for: between Supabase
15 and 17, pg_cron 1.6 against 1.6.4 failed, and the whole apply with
it. The target's own update paths now decide: a version it can reach
gets the ALTER, any other is reported with no fix and why.

An extension in a schema of its own was created into it, which plain
PostgreSQL may not have — Supabase's `extensions` — and the missing
schema failed the apply. The schema is made first; an extension the
server does not ship is still left out with it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A snapshot of a newer PostgreSQL holds what an older one has no syntax
or feature for — a VIRTUAL generated column before 18, UNIQUE NULLS NOT
DISTINCT before 15 — and in one transaction that statement rolled the
whole restore back. The snapshot's version is read from pg_dump's
header in schema.sql; restoring into an older server, such a statement
is left out and named, as one needing a missing extension is, and the
rest is restored. Into a server as new, it still fails the restore.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The checks run many queries, concurrently, over seconds or minutes, and
a change made meanwhile — a migration, a policy someone adds — was seen
by some of them and not others: the report could describe a database
that never existed.

While a scan runs, one connection to each database holds a REPEATABLE
READ, READ ONLY transaction that exports its snapshot, and each pooled
connection joins it once. A query that fails ends that connection's
transaction, which joins again next time; a definition rendered with a
temporary object is undone to a savepoint. Both are let go when the
scan ends, before anything is applied. A server that cannot export a
snapshot is read as before.

As with pg_dump, what the scan reads stays locked until it ends;
--no-snapshot reads each query on its own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hema

pg_cron is in pg_catalog on Supabase, and creating the schema an
extension goes in first wrote CREATE SCHEMA IF NOT EXISTS "pg_catalog",
which PostgreSQL refuses ("unacceptable schema name") — failing the
whole apply from Supabase to plain PostgreSQL. A system schema is
always there; only another one is made first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* is let go; a read that fails, and a definition rendered with a temporary
* object, leave the reads after them working.
*/
import { describe, it, expect, beforeAll, afterAll, afterEach } from 'vitest'
jasdeepkhalsa and others added 2 commits October 18, 2026 13:30
An extension in a schema of its own makes that schema first, so its fix
no longer began with CREATE EXTENSION, and one the target's server does
not ship — pg_graphql on plain PostgreSQL — was no longer recognised as
such: it failed the whole apply instead of being left out. The fix is
now recognised by a CREATE EXTENSION anywhere in it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From PostgreSQL 18 a NOT NULL is a constraint with a name: a table or
column renamed keeps the old one, and one made afresh gets a default.
The proof compares the migrated copy with the source's structure made
anew on the target's server, so a rename from an older source read as
its NOT NULLs missing and others unexpected, and the migration was
refused. They are now matched by the column they hold for.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jasdeepkhalsa
jasdeepkhalsa merged commit 0a185a9 into main Oct 18, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

apply: revoking grants on an object the same apply drops is reported as not applied

1 participant