fix(predicate): resolve _ROW_ID by name, not by its placeholder leaf index - #642
Open
plusplusjiajia wants to merge 1 commit into
Open
fix(predicate): resolve _ROW_ID by name, not by its placeholder leaf index#642plusplusjiajia wants to merge 1 commit into
plusplusjiajia wants to merge 1 commit into
Conversation
plusplusjiajia
force-pushed
the
fix-row-id-partition-projection
branch
7 times, most recently
from
August 1, 2026 06:26
4283de0 to
2b6fbf1
Compare
plusplusjiajia
force-pushed
the
fix-row-id-partition-projection
branch
2 times, most recently
from
August 2, 2026 05:17
63b7269 to
79e9aef
Compare
plusplusjiajia
force-pushed
the
fix-row-id-partition-projection
branch
from
August 4, 2026 08:53
79e9aef to
db2abd9
Compare
plusplusjiajia
marked this pull request as ready for review
August 4, 2026 09:19
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
_ROW_ID is not a table column, so PredicateBuilder cannot resolve it and callers hand-build the leaf with an index of their own — 0, as the existing test_with_filter_extracts_row_id_ranges does. That index is a placeholder, but Rust binds predicate leaves by index at construction and never rebinds by name, so every
index-based resolution binds the predicate to whatever field sits at that index. On a table whose first field is a partition key, WHERE _ROW_ID >= 10 pruned partitions by comparing them against row ids; on a primary-key table it was pushed down as a key conjunct and dropped key versions before the merge; a residual
_ROW_ID != 5 filtered the first user column.
Java rebinds by name (PredicateRemapper, and rowIdSafeResidualFilter for row ids); pypaimon's _change_index looks the field up by name. Rust was the only implementation with no name-level step.
Brief change log
Resolve by name. spec::is_row_id_column is the single definition, and every positional resolution consults it first: the three Predicate index-mapping primitives (which is what keeps a row id off partition keys, bucket keys and PK pushdown), residual mask evaluation, per-file remapping, scan widening, the projected
field-id set, Parquet/ORC decoder pushdown, stats pruning, and the data-evolution BLOB dependency check. DataEvolutionReader attaches _ROW_ID when a residual needs it but the caller did not project it.
Row ranges. extract_row_id_ranges returns Some(vec![]) for an empty union rather than None — "matches no rows" and "nothing extracted" are different, and reporting the latter let an enclosing AND skip the subtree and overstate its ranges. Ranges are only a superset, so ranges_represent_conjunct lets a conjunct leave the
residual only when the ranges stand in for it exactly, and row_ranges_from_filter clears a derived range when its filter is replaced.
Fail closed. Only data-evolution reads can synthesize _ROW_ID; elsewhere the predicate is unenforceable and every alternative is a silent wrong answer, so reject_row_id_filter rejects it on the read path (not the builder — TableRead is public): primary-key merges, format tables, Vortex's scanless fast path, and the Mosaic
and Blob readers. This is only for _ROW_ID — it is the only reserved name a predicate references, and the only one with no schema position at all.