Skip to content

Fix parallel Polars reads with all-missing chunks - #351

Merged
ofajardo merged 1 commit into
Roche:devfrom
ac1982:fix/sparse-polars-chunks
Oct 21, 2026
Merged

ofajardo merged 1 commit into
Roche:devfrom
ac1982:fix/sparse-polars-chunks

Conversation

@ac1982

@ac1982 ac1982 commented Oct 16, 2026

Copy link
Copy Markdown
Contributor

Parallel Polars reads fail when one worker reads only missing values for a column: that chunk has Null dtype while populated chunks have a concrete dtype, so vertical concatenation raises SchemaError. All-missing datetime chunks can also fail during epoch conversion before reaching concatenation.

Before combining Polars chunks, fill only their Null column types from populated chunks. Keep strict concatenation for actual non-null type conflicts and preserve columns that are missing everywhere. Cast datetime epoch operands to their numeric type before arithmetic so all-null datetime chunks can reach the merge.

This preserves value-label conversion and single-reader inference. Tests write real SPSS files and use spawned worker processes to check missing chunks in either order, numeric/text/date/time columns, all-missing columns, row selections, and string/categorical value labels against single-process reads.

Fixes #350.

Validation on the latest dev base, Python 3.13/Linux aarch64:

  • Polars 1.30 and 2.0: all 109 tests pass on each.
  • Pandas: 107 pass, 2 Polars-specific tests skipped; basic suite: 94 pass.
  • Typing: 50 pass; runtime types: 12 pass; HTTP integration: 1 pass.
  • The new regression tests produce 7 subcase errors on the unchanged base. Existing baseline suites pass.
  • Windows/macOS and other Python versions were not run locally. Tests use spawn to avoid forking after Polars starts threads; production multiprocessing policy is unchanged.

Developed and validated with OpenAI Codex assistance.

@ofajardo
ofajardo merged commit 69039d5 into Roche:dev Oct 21, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants