Implement native adjacent byte range coalescing in cat_ranges

[yuxin00j]

Summary

This PR implements native byte range coalescing for GCSFileSystem.cat_ranges and ExtendedGcsFileSystem.cat_ranges to optimize chunk-based workload downloads (such as PyTorch Distributed Checkpoint / DCP payload loading).

Adjacent and near-adjacent byte requests on the same object are coalesced into singular block reads natively, reducing request amplification and leveraging zero-copy memoryview slicing.

Key Changes

• Core Coalescing & Unpacking Helpers (gcsfs/core.py):

  • _coalesce_ranges: Groups and merges overlapping, contiguous (max_gap=0), and near-contiguous (gap <= max_gap) range requests for a given file, generating relative slice offsets mapped to caller indices.
  • _merge_file_ranges & _is_coalesce_enabled: Unifies conditional coalescing vs 1-to-1 range mapping.
  • _unpack_range_results: Safely populates caller results with bytes when coalescing is inactive (max_gap is None or < 0) or memoryview slices when active (max_gap >= 0).
  • Safely handles bounded and unbounded ranges (end=None), empty inputs, scalar broadcasting, and separate non-overlapping clusters.

• Auto / Dynamic max_gap Calculation (gcsfs/core.py):

  • _compute_adaptive_max_gap: Dynamically derives an optimal coalescing gap per file from that file's requested chunk lengths (5% of median chunk size, capped at 1 MB). Prevents read amplification on small ranges while optimizing throughput for large megabyte-scale tensors, and avoids cross-file median skew when mixing small metadata files and large tensor shards.
  • Enables auto coalescing via auto_max_gap=True or max_gap="auto".

• GCSFileSystem._cat_ranges (gcsfs/core.py):

  • Adds native support for coalesced byte range reads across multiple files.
  • Executes chunked reads using asyn._run_coros_in_chunks respecting batch_size.
  • Supports explicit integer max_gap, auto_max_gap=True, and max_gap="auto" (computed per file).
  • Supports both on_error="return" and on_error="raise" error modes.

• ExtendedGcsFileSystem._cat_ranges (gcsfs/extended_gcsfs.py):

  • Bulk Zonal Downloads via MRD: Connects range coalescing with Zonal (RAPID) buckets via AsyncMultiRangeDownloader (MRD).
  • Dynamic Multi-MRD Pool Concurrency: Sizes pool_size = min(max_batches, max(1, concurrency)) to stream multiple batches in parallel across pooled MRDs.
  • Per-File Adaptive max_gap for Zonal Downloads: Computes effective coalescing gap per file and applies it to MRD range batches.
  • Unified Coroutine Scheduling: Combines Zonal MRD batch downloads and Non-Zonal HTTP GETs into a single coroutine pool scheduled via asyn._run_coros_in_chunks.
  • Lazy Metadata Lookup: Queries object metadata (_info()) only when unbounded ranges require file size resolution.
  • Resource Cleanup: Ensures all acquired MRDPool instances are cleanly closed in finally blocks upon completion or failure.

Testing

• Unit Tests in gcsfs/tests/test_core.py:
  • Tests for _compute_adaptive_max_gap covering empty inputs, small chunks, 10MB chunks, 100MB chunks (capped at 1MB), and custom ratios.
  • Tests for cat_ranges with auto_max_gap=True, max_gap="auto", per-file adaptive gap isolation across mixed file sizes, scalar broadcasting, contiguous coalescing (max_gap=0), multi-file coalescing, and error modes.
• Unit Tests in gcsfs/tests/test_extended_gcsfs.py:
  • Tests for non-zonal delegation (with auto_max_gap=True and max_gap="auto").
  • Tests for zonal coalescing with MockMRD (including auto coalesced ranges, wide uncoalesced ranges, and per-file adaptive gap isolation).
  • Tests for dynamic MRD pool concurrency, chunk-based batch size limits via _run_coros_in_chunks, return types (bytes vs memoryview), and pool cleanup upon error.

[gemini-code-assist]

Code Review

This pull request implements _cat_ranges and cat_ranges in both GCSFileSystem and ExtendedGcsFileSystem to fetch multiple byte ranges efficiently, supporting range coalescing and leveraging AsyncMultiRangeDownloader for zonal buckets. It also adds comprehensive unit tests. The review feedback suggests extracting the duplicated input validation logic into a shared helper and removing an unnecessary fallback default value of 64 for the batch size.

[codecov]

Codecov Report

❌ Patch coverage is 94.68085% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.46%. Comparing base (581fd26) to head (63039ba).

Files with missing lines	Patch %	Lines
gcsfs/core.py	95.53%	8 Missing ⚠️
gcsfs/extended_gcsfs.py	93.20%	7 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1024      +/-   ##
==========================================
+ Coverage   90.21%   90.46%   +0.25%     
==========================================
  Files          16       16              
  Lines        3749     4007     +258     
==========================================
+ Hits         3382     3625     +243     
- Misses        367      382      +15     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[yuxin00j]

/gcbrun

[yuxin00j]

/gcbrun

[ankitaluthra1]

@yuxin00j Can you also add pytest based before and after microbenchmarks results in PR description done over multiple runs. Looking for consistent number here for perf improvement. Lets also validate small ranges to full range read 64 kb, 5 MB file or 8 MB file and 16 MB.

[ankitaluthra1]

Keeping effective_max_gap can lead to regression in certain scenarios:

Example Scenario:
File A requests small 100-byte metadata slices.
File B requests 50 MB checkpoint shards.
If File B dominates, effective_max_gap scales up to 1 MB. This 1 MB gap is then applied to File A, causing up to 10,000× read amplification on small files.
If File A dominates, effective_max_gap collapses to 5 bytes, preventing legitimate coalescing on File B.

Consider making it per file

[raj-prince]

Overall range coalescing seems to be the right approach, but few points to consider:

1. Do we really need coalescing on rapid bucket? Given rapid provides bidi stream and you can pipeline multiple ranges on the same stream.
2. IIUC, default batch size reduces from 1280 (ref) to 64, have we measured the impact of this default behavior change?
3. Could we split this PR into 2-3 smaller PRs (core coalescing logic, standard integration, rapid integration)? Given it's too large ~2K lines, and do see scope to make the implementation simpler and more readable.
4. IIUC, info is called to get object-size where end = None, which will cost one extra round-trip to get the size in comparison with the previous implementation where end = None mean EOF? This could be a behavior change, and should avoid until very important. I feel, it's better to keep the previous behavior and don't coalesce these ranges?

[yuxin00j]

Overall range coalescing seems to be the right approach, but few points to consider:

1. Do we really need coalescing on rapid bucket? Given rapid provides bidi stream and you can pipeline multiple ranges on the same stream.
2. IIUC, default batch size reduces from 1280 (ref) to 64, have we measured the impact of this default behavior change?
3. Could we split this PR into 2-3 smaller PRs (core coalescing logic, standard integration, rapid integration)? Given it's too large ~2K lines, and do see scope to make the implementation simpler and more readable.
4. IIUC, info is called to get object-size where end = None, which will cost one extra round-trip to get the size in comparison with the previous implementation where end = None mean EOF? This could be a behavior change, and should avoid until very important. I feel, it's better to keep the previous behavior and don't coalesce these ranges?

Sure, I'm re-evaluating this change now as from the microbenchmark, setting default concurrency to 1 in _cat_file helps improve cat_ranges latency as well.