[RLC] Case 4b — DML loses to OPTIMIZE (reverse compaction offset remap)

[sezruby]

Part of the row-level-concurrency umbrella (#3) — Case 4b (reverse direction: a concurrent DML that loses to a compaction OPTIMIZE). Full design: #7. Stacked on PR #9 (Case 4a, forward OPTIMIZE-loser); both consume the composition captured & persisted by PR #12 (write side).

What

The mirror of #9's forward reconcile: a row-level DML that loses to a concurrent compaction OPTIMIZE which removed the files it touched. Opt-in via spark.databricks.delta.optimize.conflictReconciliation.reverse.enabled (internal, default off; must be on for both the OPTIMIZE and the DML sessions — it also arms the write-time capture in PR #12 so the winner persists its composition). Instead of aborting the DML, ConflictChecker.resolveReverseOptimizeConflicts reads the winning OPTIMIZE's persisted per-source composition and remaps the DML's newly-deleted rows onto the compacted output, so the DML commits.

Composition it reads (from PR #12)

The winner's compactedInto / compactionInfo tombstone tags, parsed by the shared ConflictChecker.parseOptimizeSourceComposition into (outputPath, outputStart, liveCount) per removed source — the same parser and format #9's forward direction uses. This PR adds no write-side or tag code; the capture, the persisted format, and the parser all live in #12 / #9. Multi-run / out-of-commit / unrecognized shapes → None → abort.

Reverse mechanism

The winner OPTIMIZE commits RemoveFile(F1..Fn) + AddFile(C, dataChange=false) — required to be a pure compaction (every removed source carries a composition into one of this commit's outputs, and every added output is a referenced target; otherwise abort). For each newly-deleted physical row i of the losing DML in a removed source F (only loserDv \ loserReadDv — rows already deleted at the DML's read time were dropped from C), compute outputStart + liveRank(i) where liveRank(i) = i − |read-time DV below i|, and union it into C's deletion vector (seeded from C's existing DV); then re-point the DML's RemoveFile(F) at the winner's post-image (files keyed by path and DV). All-resolvable-or-abort. A pure compaction is a dataChange=false relocation, so the DML's read stays invariant to it.

A DML-touched, winner-removed source that is left un-remapped (e.g. a full-file delete emitting a bare RemoveFile with no re-added DV'd AddFile) forces an abort rather than silently dropping the delete.

Safety

Opt-in, default off, both directions. No composition tag (repartition shuffle emptied InputFileBlockHolder, reclustering/ZORDER, a vectorized/native write backend bypassed the capture, multi-run, or an out-of-run row) → falls back to today's abort — never wrong data.

Tests — OptimizeConflictReconciliationSuite

• forward (OPTIMIZE-loser) + parser behavior carried from [RLC] Case 4a — OPTIMIZE loses to DML (forward compaction offset remap) #9;
• reverse (DML-loser): compaction winner ⟂ DELETE/UPDATE loser → both commit, read-back correct; correct remap across an OPTIMIZE-time read gap; reverse-off aborts; winner-recorded-no-composition aborts; ZORDER winner aborts; a losing full-file delete that can't be remapped aborts (mixed full/partial).

Green: OptimizeConflictReconciliationSuite (21) + OptimizeConflictSuite (1) + RowLevelConcurrencySuite (9).

Dependency

Stacked on #9 (Case 4a, base branch optimize-dv-remap), which stacks on the capture/persist write side #12 (optimize-dv-remap-capture) and the same-file DV-merge PoC (Case 2 — #5 / PR #2). Combined design: #7.

🤖 Generated with Claude Code