[core] Support TopN pushdown for BTree global indexes - #8991
Open
discivigour wants to merge 13 commits into
Open
[core] Support TopN pushdown for BTree global indexes#8991discivigour wants to merge 13 commits into
discivigour wants to merge 13 commits into
Conversation
added 9 commits
August 2, 2026 21:48
leaves12138
previously requested changes
Aug 3, 2026
added 2 commits
August 3, 2026 11:14
discivigour
marked this pull request as ready for review
August 3, 2026 04:20
JingsongLi
requested changes
Aug 3, 2026
JingsongLi
left a comment
Contributor
There was a problem hiding this comment.
Can you add tests for Spark SQL?
Contributor
|
Please disable global-index TopN for incremental scan domains. |
Contributor
Author
Added. |
Contributor
Author
Thanks, I will change it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Why
BTree global indexes could evaluate predicates, but TopN planning could not use the index ordering to reduce index and data reads. Evaluating each file independently is insufficient because the final candidates must be compared globally, and coverage handling must not treat intentionally pruned index files as unindexed data.
This change computes a conservative global candidate set for supported single-column TopN queries. The execution engine still retains the final sort and limit, so the pushdown remains a pruning optimization rather than the source of final result ordering.
Impact
Data-evolution tables with a compatible BTree global index can reduce index-file opens, index-entry scans, and downstream row reads for ascending or descending TopN queries.
Validation
mvn -pl paimon-core -am \ -Dmaven.repo.local=/opt/homebrew/opt/maven/repository \ -DwildcardSuites=none \ -DfailIfNoTests=false \ -Dtest=GlobalIndexEvaluatorTest,TopNGlobalIndexResultTest,BTreeIndexReaderTest,LazyFilteredBTreeIndexReaderTest,BlockIteratorTest,BTreeTopNIndexFileSelectorTest,BtreeGlobalIndexTableTest \ testgit diff --checkpassed