[devteamaegis]

read_file_multiprocessing splits the rows to read across num_processes workers. When fewer rows are requested than there are processes, some workers get a share of 0 rows. That 0 is passed to the read function as row_limit=0, which means "no limit", so those workers read every remaining row and the result has many duplicated rows.

Example with test_data/basic/sample_large.sav (485 rows):

df, _ = pyreadstat.read_file_multiprocessing(pyreadstat.read_sav, f, num_processes=4, row_limit=2)
len(df)  # 968, expected 2

The same happens with read_file_in_chunks(..., multiprocess=True) when chunksize is smaller than num_processes, and with read_file_multiprocessing(..., row_limit=0), which returned num_processes copies of the file.

The fix skips chunks with 0 rows. If all chunks are empty (empty file, or offset at or past the end), one is kept so an empty dataframe with the right columns is still returned, as before.

I added a test for each case to test_basic.py and test_narwhalified.py. They fail on dev and pass with the fix. The full test_basic, test_narwhalified (pandas and polars) and test_runtime_types suites pass.

Row counts on sample_large.sav (485 rows) with num_processes=4:

Call	dev	This PR
row_limit=2	968	2
row_limit=0	1940	485
default	485	485

With this change, an explicit row_limit=0 reads the file in a single worker. If you'd like it to stay parallel, treating row_limit=0 as float("inf") at the top of the function would do it; I kept this PR to one change.

[ofajardo]

Good catch and good fix, thanks!

Tests are failing due to introduction of polars 2.0 I just fixed it on dev. Therefore, please update your branch with the latests changes on dev. If all tests pass here on github, then I merge.

[devteamaegis]

Thanks! I merged the latest dev into the branch. The CI run is waiting for approval to run since this comes from a fork. Locally on the updated branch, test_basic.py (94) and test_narwhalified.py with pandas and polars (107 each) all pass.

[ofajardo]

Thanks!