Repository navigation
Conversation
Append-only consumption rows, the --changed decision over recent executions, and a seeded simulation whose oracle tracks what every model read. Validated so far: the core crate's tests and clippy only; nothing is wired into builds yet.
Adapters gain consumption-table DDL, pruning and a UTC warehouse clock; the build records started and succeeded rows and --changed skips unchanged models. Validated so far: cargo check and a local DuckDB smoke run only.
… docs Consumption I/O moves behind a native-host edge, scheduled jobs accept the changed setting and report skipped models as successful, the janitor prunes the table, and the build docs describe --changed. Validated locally: make check-ci, cargo tests for sqlbuild-state, DuckDB and SQLite end-to-end CLI, crash and scheduler tests, and manual PostgreSQL, ClickHouse and SQL Server recorded_at checks. Not yet reviewed; timings pending.
…etween phases Mirror _sqlbuild_consumption into the control store through the existing warehouse-history sync (new history kind, ingest, route, client and an additive migration). --changed never reads the mirror. The recorder now connects only for its read and append phases instead of holding a warehouse connection for the whole build, so ingestr loads and in-process crash/retry paths can open a file-backed DuckDB warehouse. Janitor e2e expectations and the planner differential masks account for the new always-written table. Validated locally: make check-ci, Fensu, hygiene scan; janitor e2e 43/43; ingestr e2e 6/6; migration crash/retry integration 56/56; planner differential; changed-builds e2e, scheduler --changed with mirror, and warehouse-metadata integration 43/43; orchestration native tests for the mirror on SQLite and PostgreSQL 7/7; TypeScript client check and test. Timings and independent review still pending.
The undeclared-read guard parsed model SQL with function reference
calls still in it. The parser cannot read __udf("name")(...), so every
model calling a SQL or table function, and everything downstream, ran
on each --changed build. Function calls now parse as plain calls; what
a function reads is already a declared input.
Also drops the two slow served-location consumption cases from the
warehouse metadata integration test, which pushed the CI job past its
timeout. The mirror stays covered by the native orchestration test and
the scheduler --changed test.
WIP: validated locally with make check-ci, the hygiene scan, the
changed-builds e2e, warehouse metadata and scheduler --changed tests,
and a 3,000-model --changed timing run.
Function call stubbing let the undeclared-read guard parse models that call SQL functions, but it only checked the model's own SQL. A table function or SQL function whose body read a relation it does not declare was invisible, so the model calling it could be skipped while stale. The guard now also checks the body of every SQL function a model calls, transitively, against that function's declared inputs. A body that reads anything else, fails to parse, or belongs to an unknown function forces the model to run. Python functions read no relations. The DuckShop changed_builds_udf case gains a table function and a SQL function that read raw.customers undeclared, and the DuckDB e2e test checks that their models run on every --changed build and pick up a new customer. WIP: validated locally with make check-ci, the hygiene scan, and the changed-builds e2e and scheduler --changed tests.
The undeclared-read guard accepted a relation name the SQL spells without its schema or database whenever a declared input had that name. When two project relations share the name across schemas, the warehouse resolves the omitted qualifier from its session default and can read the one that is not declared, so --changed skipped a stale model: a model declaring the raw customers source while reading the customers model unqualified did not rerun after that model changed. A name that matches more than one project relation now forces the model to run. Project relations are indexed by name, so each check stays constant time. The DuckShop changed_builds_unqualified case reproduces it, and a DuckDB e2e test checks that the reader runs and picks up the change. WIP: validated with make check-ci, the hygiene scan, the changed-builds e2e and scheduler --changed tests, and paired 3,000-model timings.
The reader does not declare the customers model, so nothing orders it after that model's rebuild and it can read the old rows, as it did on CI. It runs on every --changed build, so the test now checks its data after the next build, once customers has settled. WIP: validated with make check-ci, the hygiene scan and the changed-builds e2e tests.
The base now syncs every warehouse history kind through one endpoint and keeps five typed mirror tables, which their UI readers will use. The consumption mirror this branch added had no reader in the server or the UI, so it is not carried over: its table, ingest path, route, client method, sync kind and mirror tests go. Changed builds still read consumed inputs from the warehouse's own consumption table.
The decision rules were a trait with one real implementation and four broken ones that only the simulation test used. The shipped rules are now plain code. The simulation still proves its oracle finds misses by writing consumption without the started row.
Appends now go in fixed-size statements: keeping an execution inside one statement bought nothing, because separate statements are not atomic together and the decision already treats a partly written execution as unfinished. Source freshness and consumption share one state-table naming helper, and seeds use the same forced-run reasons as models. The consumption connection's close joins the allowlist of context objects that each release their own resource.
…e place Each SQL Server state index repeated the sys.indexes guard, and each keyed state table repeated the NVARCHAR(450) key narrowing. Postgres repeated CREATE INDEX IF NOT EXISTS per table. One private builder per adapter now takes the table, index and columns, and the fingerprint, freshness, node-result, audit-result and consumption methods use it. The rendered SQL is unchanged.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
sqb build --changedshould build only models and seeds whose definition or inputs changed since they last succeeded, with no control store and no missed run after crashes, retries or partial selections.Changes
_sqlbuild_consumptionper target schema: astartedrow before each write, thensucceededrows with each direct input's freshness hash or upstream execution ID.sqb build --changedand the jobchangedsetting (off unless the job sets it; no project key) decide from those records only. Unknown or undeclared reads (also in called functions, or ambiguous names), unfinished runs, planner changes, full refresh and cursor overrides always run.recorded_at, janitor prunes whole executions.--changednever reads it.changed_builds,changed_builds_udf,changed_builds_unqualified,consumption_history; docs for--changed, janitor and retention.Verification
recorded_aton DuckDB, SQLite, PostgreSQL, ClickHouse, SQL Server.--changedno-op 21.9 s (0 of 3,141 run); one edit 23.3 s (3 run), as before.make check-ci, hygiene scan pass.