Skip to content

Close underlying files when using compression - #2120

Open
Jokasa7 wants to merge 1 commit into
fsspec:masterfrom
Jokasa7:JoyCx/fix-compressed-file-close
Open

Jokasa7 wants to merge 1 commit into
fsspec:masterfrom
Jokasa7:JoyCx/fix-compressed-file-close

Conversation

@Jokasa7

@Jokasa7 Jokasa7 commented Sep 7, 2026

Copy link
Copy Markdown

Closing a file returned by AbstractFileSystem.open(..., compression=...) can leave the underlying filesystem file open. For codecs such as gzip, bz2 and lzma, closing the compression stream does not close a caller-provided file. This delays resource release and can defer a write/upload error until garbage collection.

Keep ownership of both streams in a private wrapper. Close the compression stream first to finish its trailer, then close the underlying file in finally. Preserve the unwrapped object when compression inference selects no codec, and keep transaction queues pointing to the raw file.

The returned object retains the file-like interface but is a wrapper, so its concrete type is no longer the codec class. This avoids replacing methods on codec objects, including C extension streams that do not allow setting close.

Fixes #1672.

Validation on Python 3.12.13 / Windows:

  • The original 30 new regression cases fail against unmodified master: 24 leave the raw file open and six fail to propagate its close error.
  • The final compression module tests pass: 68 passed, 1 skipped (optional snappy dependency unavailable). Coverage includes binary/text reads and writes, direct/context-manager close, compression/raw close failures, transaction commit/discard, iteration/seek/readinto, no-codec identity, and codecs that already close their input.
  • The affected test set reports 491 passed, 135 skipped, 10 xfailed and 2 failed. Both failures are existing test_glob_weird_characters cases that try to create a directory containing |, which Windows rejects with WinError 123; the same failures were reproduced on unmodified master.
  • Ruff check/format, git diff --check, and a full Sphinx HTML build with -W --keep-going pass. The optional full cloud/backend and downstream integration suites were not run locally.

AI assistance: this patch and its regression tests were developed and validated with OpenAI Codex, including a separate code review. The test results above are from commands executed by Codex.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Context not preserved when open file with compression, leading exception to be ignored

1 participant