[RLC] Case 4a — OPTIMIZE loses to DML (forward compaction offset remap)

[sezruby]

Part of the row-level-concurrency umbrella (#3) — Case 4a (forward direction: a compaction OPTIMIZE that loses to a concurrent DML). Full design: #7. The shared capture/persist write side is PR #12 (this stacks on it); the reverse DML-loser direction is PR #11 (stacked on this).

What

Opt-in reconciliation (spark.databricks.delta.optimize.conflictReconciliation.enabled, internal, default off) for a compaction OPTIMIZE that loses to a concurrent row-level DELETE/UPDATE. Instead of aborting the loser, ConflictChecker.resolveOptimizeConflicts reads the source composition PR #12 persisted, remaps the concurrent deletion vector from each removed source onto the compacted output, unions it into that output's DV, and re-points the source RemoveFile at the winner's post-image (files keyed by path and DV) — so both commit.

Mechanism

Compaction concatenates each source file's live rows into one contiguous run in the output, so the reconcile reduces to where each source row landed. From the persisted compactedInto / compactionInfo tags, ConflictChecker.parseOptimizeSourceComposition recovers (outputPath, outputStart, liveCount) per removed source. For each of the winner's newly-deleted physical rows it computes the output position, unions it into the output DV, and commits both. All-resolvable-or-abort — any row that can't be placed falls back to today's abort.

Offset remap with lazy read-time-gap reconstruction

A source that was fully live at read time maps physical row i → outputStart + i. A source that carried a read-time DV has physically non-contiguous live rows, but the tag records only its physical count — the gaps are reconstructed lazily at conflict time:

liveRank(i) = i − (# read-time deletions below i)     // binary search on the sorted read-time DV
outputPos   = outputStart + liveRank(i)

Only the winner's incremental deletions (winnerDv \ readTimeDv) are remapped; positions already deleted at read time were never written to the output and are skipped. A row whose liveRank lands outside the source's recorded run → abort.

Scope / guards

Reconciles only what PR #12 captured (order-preserving coalesce compaction). Reclustering / ZORDER and the repartition path record no composition → this falls back to today's abort, which matches the documented RLC contract (a non-ZORDER OPTIMIZE "cannot conflict" — here it simply doesn't, by reconciling). Safe because opt-in: losing the reconcile restores current behavior, never wrong data.

Dependency

Stacked on PR #12 (Case 4 capture/persist, base branch optimize-dv-remap-capture), itself stacked on the same-file DV-merge PoC (Case 2 — #5 / PR #2). Reuses that PoC's DV helpers and rowLevelResolvedPaths.

Tests — OptimizeConflictReconciliationSuite (forward + parser)

Reconcile behavior:

• reconciles a concurrent DELETE / UPDATE by remapping the DV onto the compacted output;
• reconciles a DELETE when the source already carried a read-time DV (lazy gap reconstruction), incl. multiple gaps in one source;
• remaps a delete to a non-DV file in a bin that also holds a DV'd file;
• targets the correct output among multiple compaction bins;
• partitioned compaction reconciles, and the capture reports the partition ordering (no sort above the capture);
• reclustering (ZORDER) and repartition-path losers are not reconciled and abort.

Parser (composition tag:): single-run parse, live = physical − read-time-DV, tolerates unknown fields (schema drift), falls back to None on shapes the contiguous offset remap can't model.

🤖 Generated with Claude Code