Repository navigation
Add DuckDB backend - #42
Merged
Merged
Conversation
…nd reinsert, update CI, smaller helpers
nicoloesch
added
the
feature
This was referenced Oct 16, 2026
create_tables() only consulted resolved's static schema-translate map when deciding if a foreign key can span two tags, so a tag claimed only via create_engine(schema_claims=[...]) raised UnregisteredSchemaTagError even though the connection itself already resolves it.
gkennos
self-requested a review
October 27, 2026 00:07
gkennos
requested changes
Oct 27, 2026
…fore each batch commit - deferred_fk_constraints/check_replace_feasible now require resolved and delegate to unenforced_foreign_keys() instead of a local cyclic-only approximation, closing the cross-schema blind spot - validate deferred FKs before every intermediate batch commit, not just once after the whole merge, so an invalid batch is never committed in the first place
gkennos
requested changes
Oct 27, 2026
…e FKs for real - merge_insert/merge_upsert load a self-referential deferred FK wave by wave instead of by insertion order, so a child batched before its own parent no longer false-positives, while a genuine orphan is still caught before commit - DuckDBBackend.validate_deferred_fks checks a cross-database FK by querying each connection separately and comparing in Python, instead of skipping it
…n dependency order Replace and insert ran as two sealed phases, so an existing row's self-referential FK could be re-pointed at a row only the later insert phase adds, validating it before that row exists. merge_replace now updates and inserts together through the same dependency-ordered wave loop, draining staging itself so the merge_insert() call that follows has nothing left to do.
…alidation Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
gkennos
self-requested a review
October 27, 2026 23:25
DuckDBBackend now overrides _paginate_batches to yield self-referential batches in dependency order, falling back to the base class's ordinary pagination otherwise. This replaces _merge_in_dependency_waves, fixing unbatched loads committing per wave, valid self-links/cycles being rejected as orphans, validate_deferred_fks=False being ignored by an unconditional final check, and upsert's conflict handling being shadowed by premature readiness checks. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…tion _validate_cross_database_fk rebuilt the remote role's engine from config, which silently gets an empty, disconnected database for a DuckDB ':memory:' connection instead of reconnecting to real data. Raise clearly instead, matching bootstrap.py's existing is_ephemeral_url guard for the same failure mode. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
_pagination_column/_ensure_pagination_index were byte-for-byte identical in postgres.py and duckdb.py. Make them the base class's concrete default (ordering by staging's _rownum, indexed on demand), with SQLite's rowid-based override now explicit about why it skips indexing instead of relying on the old no-op default. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
gkennos
reviewed
Oct 28, 2026
gkennos
left a comment
gkennos
left a comment
Member
There was a problem hiding this comment.
the earlier merge and unbatched-commit issues are fixed. The real-vocabulary test loaded 40,466,062 rows (stubbed temporary null handling).
Queries, CRUD, FK validation and maintenance checks passed. No further changes requested for those resolved loader issues.
One thing to note is that restore pathway does commit all tables in one go which means it maxes RAM sooner than you'd otherwise expect but also it fails sensibly so just for noting...
gkennos
self-requested a review
October 28, 2026 11:45
gkennos
approved these changes
Oct 28, 2026
gkennos
left a comment
gkennos
left a comment
Member
There was a problem hiding this comment.
sorry meant to approve but accidentally only commented :)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Changes
duckdbextra, with staging tables and batchedreplace,upsertandinsertmerges.deferred_fk_constraints().load_csv(validate_deferred_fks=True)checks those keys with an anti-join after each merge.replaceupdates rows in place, because DuckDB rejects deleting a referenced row.check_replace_feasible()lets callers reject an impossiblereplacebefore loading anything.replacewithindex_strategy="keep"is rejected up front on backends that can't support it._paginate_batcheshelper instead of a copy per backend.load_csvalways drops its staging table, including when the merge fails.Checklist
breaking,feature,fix,dependencies, orchore)uv run pytest -q)uv run ruff check .)