Repository navigation
Fix parallel Polars reads with all-missing chunks - #351
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Parallel Polars reads fail when one worker reads only missing values for a column: that chunk has
Nulldtype while populated chunks have a concrete dtype, so vertical concatenation raisesSchemaError. All-missing datetime chunks can also fail during epoch conversion before reaching concatenation.Before combining Polars chunks, fill only their
Nullcolumn types from populated chunks. Keep strict concatenation for actual non-null type conflicts and preserve columns that are missing everywhere. Cast datetime epoch operands to their numeric type before arithmetic so all-null datetime chunks can reach the merge.This preserves value-label conversion and single-reader inference. Tests write real SPSS files and use spawned worker processes to check missing chunks in either order, numeric/text/date/time columns, all-missing columns, row selections, and string/categorical value labels against single-process reads.
Fixes #350.
Validation on the latest
devbase, Python 3.13/Linux aarch64:Developed and validated with OpenAI Codex assistance.