Skip to content

GH-3719: Fix vectored read allocation limits and fallback safety - #3726

Open
sunchao wants to merge 1 commit into
apache:masterfrom
sunchao:dev/chao/codex/gh-3719-vectored-read-safety
Open

GH-3719: Fix vectored read allocation limits and fallback safety#3726
sunchao wants to merge 1 commit into
apache:masterfrom
sunchao:dev/chao/codex/gh-3719-vectored-read-safety

Conversation

@sunchao

@sunchao sunchao commented Aug 16, 2026

Copy link
Copy Markdown
Member

Why are the changes needed?

Apache Parquet's Hadoop vectored-read path can fall back to ordinary reads after
asynchronous filesystem requests have already been submitted or partially
consumed. Retrying against the same partially populated chunk builder can
duplicate selected page data and return incorrect filtered rows. Outstanding
sibling requests can also continue using a stream while fallback or cleanup
begins.

The same path requests one buffer for an entire contiguous projected range
instead of splitting requests at parquet.read.allocation.size, which can cause
unexpectedly large allocations.

Vectored I/O is enabled by default in Apache Parquet 1.16 and later, so these
issues are not limited to applications that explicitly opt in.

What changes were proposed in this PR?

  • Split contiguous requested filesystem ranges at the configured allocation
    limit while preserving the existing logical page and column plan.
  • Keep ordinary fallback for failures detected while preparing vectored ranges;
    convert failures after the vectored submission call is attempted into
    IOException rather than replaying reads against a partially populated
    builder.
  • Await already-published sibling read futures within the existing shared
    timeout before propagating an asynchronous failure.
  • Verify dictionary pages and V1/V2 data-page checksums incrementally across
    split buffers instead of allocating another oversized contiguous buffer.
  • Add focused vectored-reader and checksum regression coverage.

The current patch still has four known review points that need follow-up:

  • Filesystems that align ranges for checksums can allocate more than the
    requested range size, so splitting requests alone does not guarantee that
    every backend allocation respects the configured cap.
  • A backend that rejects vectored I/O before submitting work should retain safe
    ordinary-read fallback; the current submission boundary cannot distinguish
    that case from a partially submitted failure.
  • Filesystems such as S3A can block while synchronously submitting more ranges
    than their in-flight limit permits, before the existing read timeout applies.
  • A backend can start reads and then fail before publishing their futures to
    the wrapper, leaving those reads invisible to the current cleanup path.

How was this PR tested?

Previously recorded results for this commit on Java 17:

mvn -pl parquet-hadoop \
  -Dtest=TestParquetFileReaderVectoredIO,TestDataPageChecksums \
  -Dsurefire.failIfNoSpecifiedTests=false \
  test

The focused suites reported 37 passing tests, including 19 vectored-reader
tests
and 18 checksum tests.

mvn -pl parquet-hadoop test
mvn -pl parquet-hadoop spotless:check

The complete module reported 748 tests, zero failures, zero errors, and 24
existing Hadoop-capability skips
. Apache RAT approved all 256 scanned
licenses, and Spotless passed. The four review points above are not covered by
those recorded passing tests.

Closes #3719.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Vectored Parquet reads can fall back unsafely after partial asynchronous reads and exceed allocation limits

1 participant