[Spark] Handle merge-on-read re-adds in the delete/read conflict refinement (RLC Case 1b) - #16
Closed
sezruby wants to merge 3 commits into
Closed
Conversation
The delete/read conflict check aborts the current transaction whenever a concurrently-removed file's path is in its read set, regardless of whether the removed rows match what it read. Behind conflictDetection.deleteReadDataSkipping.enabled (default off), refine this to a row-level check: read the rows the winner actually removed from the overlapping files and conflict only when a removed row matches the current transaction's read predicates. The removed rows are obtained without an inverse deletion-vector read. Because a DML only ever adds deletions, the winner's post-image deletion vector is a superset of the pre-image one, so matches(removed) = matches(pre-image live view) - matches(post-image live view) -- two ordinary reads of the (few) overlapping files suffice, with no per-file attribution. This is the delete/read analogue of the value-exact added-files skipping. One-way safe and fail-safe: the loser aborts unless the removed rows are proven not to match; any non-applicable case, missing information, or error keeps today's path-keyed abort. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
anyRemovedRowMatchesReadPredicate computed matches(removed) as countMatchingLiveRows(preImage) - countMatchingLiveRows(postImage), issuing two separate count() actions (two Spark jobs) during conflict detection. Union the two images with signed weights (+1 pre-image, -1 post-image) and sum once, so the signed difference is produced in a single job. A full-file removal has no post-image, so the pre-image matches alone are the removed-row count. Behavior is unchanged (same count-difference, same fail-safe); only the number of conflict-time jobs drops from two to one. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…nement A merge-on-read DELETE removes rows by widening a file's deletion vector: it emits RemoveFile(P, oldDV) AND re-adds AddFile(P, newDV). The re-added file's live rows are the survivors that already existed in the current transaction's read snapshot -- it carries no new data. But the added-files conflict check treats it like any newly-appended file, so a reader whose predicate is disjoint from the removed rows (yet overlaps the survivors) aborts with ConcurrentAppendException before the row-level delete/read refinement is ever consulted. Exclude such re-adds (a path present in both the winner's added and removed sets) from the added-files check: they can only ever raise a delete/read conflict, which is resolved row-level in the delete/read check. Genuinely new rows from an UPDATE/MERGE land at a fresh path (not in removedFiles) and remain checked. With this, a reader disjoint from the removed rows commits while an overlapping reader still aborts with delete/read. Both the re-add exclusion and the delete/read row-level refinement read data during conflict detection, so they ride on the existing value-exact flags (conflictDetection.dataSkipping.enabled + conflictDetection.dataSkipping.valueExact.enabled) instead of a separate default-off config; the standalone conflictDetection.deleteReadDataSkipping flag is removed and its documentation folded into the value-exact flag. Adds two end-to-end merge-on-read tests (disjoint reader commits, overlapping reader aborts with delete/read) and updates the suite to the shared flags. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
sezruby
force-pushed
the
conflict-data-skipping-delete-read
branch
from
August 19, 2026 23:16
f9969af to
9471388
Compare
Owner
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #15. Base is
conflict-data-skipping-delete-read, so this diff is only the companion work; review #15 first.Why
#15 refines the delete/read arm so a concurrently-removed file the transaction read conflicts only when a removed row actually matches the read predicate. But for a merge-on-read DELETE that alone is not enough to let a disjoint reader commit.
A MoR DELETE removes rows by widening a file's deletion vector: it emits
RemoveFile(P, oldDV)and re-addsAddFile(P, newDV). The re-added file's live rows are the survivors that already existed in the transaction's read snapshot — it carries no new data. The added-files check, however, treats it like any newly-appended file, so a reader whose predicate is disjoint from the removed rows (yet overlaps the survivors) aborts withConcurrentAppendExceptionbefore the delete/read refinement is ever consulted.Empirically, with #15's refinement on but this fix absent, a disjoint reader racing
DELETE ... WHERE id < 10:id >= 50(disjoint from removed[0,10)) →ConcurrentAppendException(spurious)id < 5(overlaps removed rows) →ConcurrentDeleteReadException(correct)What
Exclude re-adds from the added-files check. A path present in both the winner's added and removed sets is a MoR modification, not new data; it can only raise a delete/read conflict, which the delete/read check now resolves row-level. Genuinely new rows from an UPDATE/MERGE land at a fresh path (not in
removedFiles) and remain checked. In-memory path-set filter — no extra Spark job. With this, the disjoint reader above commits; the overlapping reader still aborts with delete/read.Fold the refinement under the value-exact flags. Both the re-add exclusion and the delete/read refinement read data during conflict detection, so they gate on the existing
conflictDetection.dataSkipping.enabled+conflictDetection.dataSkipping.valueExact.enabledrather than a separate default-off config. The standaloneconflictDetection.deleteReadDataSkipping.enabledflag is removed and its docs folded into the value-exact flag.Single conflict-time job.
anyRemovedRowMatchesReadPredicatepreviously issued twocount()actions (two jobs) formatches(pre) - matches(post). It now unions the two images with signed weights (+1 pre, -1 post) and sums once — one job.Tests
Adds two end-to-end MoR tests (disjoint reader commits; overlapping reader aborts with delete/read) and moves the suite to the shared flags. Local: Case 1b 11/11, Case 1 value-exact 16/16,
OptimisticTransactionSuite143/143, scalastyle clean.Follow-up
A single umbrella flag turning on all of Case 1 + Case 1b together, eventually default-on, is left for a later change.
🤖 Generated with Claude Code