GH-3719: Fix vectored read allocation limits and fallback safety - #3726
Open
sunchao wants to merge 1 commit into
Open
GH-3719: Fix vectored read allocation limits and fallback safety#3726sunchao wants to merge 1 commit into
sunchao wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why are the changes needed?
Apache Parquet's Hadoop vectored-read path can fall back to ordinary reads after
asynchronous filesystem requests have already been submitted or partially
consumed. Retrying against the same partially populated chunk builder can
duplicate selected page data and return incorrect filtered rows. Outstanding
sibling requests can also continue using a stream while fallback or cleanup
begins.
The same path requests one buffer for an entire contiguous projected range
instead of splitting requests at
parquet.read.allocation.size, which can causeunexpectedly large allocations.
Vectored I/O is enabled by default in Apache Parquet 1.16 and later, so these
issues are not limited to applications that explicitly opt in.
What changes were proposed in this PR?
limit while preserving the existing logical page and column plan.
convert failures after the vectored submission call is attempted into
IOExceptionrather than replaying reads against a partially populatedbuilder.
timeout before propagating an asynchronous failure.
split buffers instead of allocating another oversized contiguous buffer.
The current patch still has four known review points that need follow-up:
requested range size, so splitting requests alone does not guarantee that
every backend allocation respects the configured cap.
ordinary-read fallback; the current submission boundary cannot distinguish
that case from a partially submitted failure.
than their in-flight limit permits, before the existing read timeout applies.
the wrapper, leaving those reads invisible to the current cleanup path.
How was this PR tested?
Previously recorded results for this commit on Java 17:
mvn -pl parquet-hadoop \ -Dtest=TestParquetFileReaderVectoredIO,TestDataPageChecksums \ -Dsurefire.failIfNoSpecifiedTests=false \ testThe focused suites reported 37 passing tests, including 19 vectored-reader
tests and 18 checksum tests.
mvn -pl parquet-hadoop test mvn -pl parquet-hadoop spotless:checkThe complete module reported 748 tests, zero failures, zero errors, and 24
existing Hadoop-capability skips. Apache RAT approved all 256 scanned
licenses, and Spotless passed. The four review points above are not covered by
those recorded passing tests.
Closes #3719.