[Spark] Conflict-time data skipping (delete/read tier) - #15
Open
sezruby wants to merge 1 commit into
Open
Conversation
sezruby
force-pushed
the
conflict-data-skipping-delete-read
branch
from
August 19, 2026 20:41
cfc2f36 to
f9969af
Compare
sezruby
force-pushed
the
conflict-data-skipping-delete-read
branch
4 times, most recently
from
August 25, 2026 20:13
1d396de to
2559729
Compare
sezruby
changed the title
sezruby
force-pushed
the
conflict-data-skipping-delete-read
branch
from
August 25, 2026 21:41
2559729 to
7e322e9
Compare
sezruby
changed the title
sezruby
force-pushed
the
conflict-data-skipping-delete-read
branch
from
August 25, 2026 21:42
7e322e9 to
26623f0
Compare
The delete/read check (checkForDeletedFilesAgainstCurrentTxnReadFiles) is path-keyed: it aborts the current transaction whenever a file the winner removed is in the transaction's read set, regardless of whether any removed ROW matches what the transaction read. This refines it to a row-level check, the delete/read analogue of the value-exact added-files skipping. Remove arm. Of the removed files the transaction read, keep the abort only if the winner removed a row matching a read predicate. The removed rows are obtained without an inverse deletion-vector read: because a DML only ever ADDS deletions, the winner's new DV is a superset of the pre-image one, so matches(removed) = matches(pre-image live) - matches(post-image live) where the pre-image is the removed file under its old DV and the post-image is the winner's paired re-added file under its new DV (absent for a full-file removal). The two images are unioned with signed weights (+1 pre, -1 post) and summed in a single Spark job; the difference is >= 1 iff some removed row matches. New reader ConflictDataSkippingReader.anyRemovedRowMatchesReadPredicate, wired in via the small private helper removedRowMatchesReadPredicate; the existing path-keyed lookup is unchanged when the refinement is off. Add arm. A merge-on-read DELETE removes rows by widening a file's DV: it emits RemoveFile(P, oldDV) AND re-adds AddFile(P, newDV). The re-add's live rows are survivors that already existed in the read snapshot -- it carries no new data, so it is excluded from the added-files check (checkForAddedFilesThatShouldHaveBeenReadByCurrentTxn) by an in-memory path filter. Without this, a reader disjoint from the removed rows would still abort on the added-files arm with ConcurrentAppendException before the remove-arm refinement is consulted. Genuinely new rows from an UPDATE/MERGE land at a fresh path (not in removedFiles) and remain checked. The value-exact scan of getFirstFileMatchingPartitionPredicates is otherwise untouched. Both arms read data during conflict detection, so they ride on the existing value-exact flags (conflictDetection.dataSkipping.enabled + conflictDetection.dataSkipping.valueExact.enabled) rather than a separate config (deleteReadRowLevelRefinementEnabled). One-way safe (abort unless a scan proves no removed row matches) and fail-safe (any error or missing information keeps the path-keyed abort). The remove-arm scan runs as its own Spark job, launched on a shared DeltaThreadPool just before the added-files scan so the two independent jobs overlap; the delete/read check harvests the future (ThreadUtils.awaitResult), falling back to inline execution when no scan was started. The checker-side glue lives in a self-typed trait ConflictCheckerDataSkipping mixed into ConflictChecker, mirroring the reader-side ConflictDataSkippingReader, so the feature stays a single additive unit. Tests: new DeleteReadConflictDataSkippingSuite (11) -- direct-reader count tests with real DVs, whole-file COW delete e2e, and merge-on-read e2e (disjoint reader commits, overlapping reader aborts with delete/read). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
sezruby
force-pushed
the
conflict-data-skipping-delete-read
branch
from
August 26, 2026 18:16
26623f0 to
f3b6a00
Compare
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Refines the delete/read conflict check (
checkForDeletedFilesAgainstCurrentTxnReadFiles) from path-keyed to row-level — the delete/read analogue of the value-exact added-files skipping. Stacked on the value-exact PR (base branchconflict-data-skipping-value-exact).Today the check aborts the current transaction whenever a file the winner removed is in the transaction's read set, regardless of whether any removed row matches what the transaction actually read. This refines it so a delete/read conflict is raised only when a removed row matches a read predicate.
How
Remove arm. Of the removed files the transaction read, keep the abort only if the winner removed a row matching a read predicate. The removed rows are recovered without an inverse deletion-vector read: because a DML only ever adds deletions, the winner's new DV is a superset of the pre-image's, so
where the pre-image is the removed file under its old DV and the post-image is the winner's paired re-added file under its new DV (absent for a full-file removal). The two images are unioned with signed weights (
+1pre,-1post) and summed in a single Spark job; the difference is>= 1iff some removed row matches. New readerConflictDataSkippingReader.anyRemovedRowMatchesReadPredicate, wired in via a small private helper; the existing path-keyed lookup is unchanged when the refinement is off.Add arm (merge-on-read companion). A merge-on-read DELETE removes rows by widening a file's DV: it emits
RemoveFile(P, oldDV)and re-addsAddFile(P, newDV). The re-add's live rows are survivors that already existed in the read snapshot — it carries no new data, so it is excluded from the added-files check (checkForAddedFilesThatShouldHaveBeenReadByCurrentTxn) by an in-memory path filter. Without this, a reader disjoint from the removed rows would still abort on the added-files arm withConcurrentAppendExceptionbefore the remove-arm refinement is consulted. Genuinely new rows from an UPDATE/MERGE land at a fresh path (not inremovedFiles) and remain checked. The value-exact scan itself is otherwise untouched.Config
Both arms read data during conflict detection, so they ride on the existing value-exact flags —
conflictDetection.dataSkipping.enabled+conflictDetection.dataSkipping.valueExact.enabled— rather than a separate config. The refinement is one-way safe (abort unless a scan proves no removed row matches) and fail-safe (any error or missing information keeps today's path-keyed abort).Tests
New
DeleteReadConflictDataSkippingSuite(11): direct-reader count tests with real deletion vectors, whole-file COW delete e2e, and merge-on-read e2e (disjoint reader commits, overlapping reader aborts with delete/read).🤖 Generated with Claude Code