Conversation
pd.DataFrame.empty returns True for 0-row DataFrames even when columns exist, so transform() took the else branch and returned only the numerical columns, missing all one-hot encoded columns. sklearn's _set_output wrapper then raised ValueError: Length mismatch when assigning get_feature_names_out() to the narrower DataFrame. Replace the 'not categorical_data.empty' check with an explicit 'categorical_columns and len > 0' guard and add an elif branch that builds an empty DataFrame with the correct one-hot encoded column schema when 0 rows are present, keeping the pipeline column contract stable. Strengthen test_empty_dataframe_handling and add test_transform_zero_rows_preserves_one_hot_columns regression test.
…N at inference During fit, all-NaN columns are dropped from the tracked column lists. During transform there was no symmetric handling: if a tracked column became entirely NaN (e.g. a column missing from inference data and filled with NaN by the caller), forward-fill could not fill it and dropna(how='any') dropped every row, producing the 0-row DataFrame that triggered the CategoricalEncoder column-mismatch error. Fit a package Imputer on the surviving columns and store per-column training statistics (mean for numerical, most_frequent for categorical). During transform, after forward-fill, columns that are still entirely NaN are filled with their stored training statistic so only genuinely unfillable rows (gap exceeded ffill_limit) are dropped by dropna. Add test_transform_column_becomes_all_nan_keeps_rows and test_transform_categorical_column_becomes_all_nan_filled_with_mode regression tests covering numerical and categorical fallback paths.
Imputer.transform() used 'not numerical_data.empty' / 'not categorical_data.empty' to guard SimpleImputer.transform() calls. pd.DataFrame.empty is True for 0-row DataFrames even when columns exist, so on 0-row input the imputation was skipped and a 0-column DataFrame was produced, relying on a downstream reindex to recover the column schema. Replace the .empty checks with explicit 'columns and len > 0' guards and add elif branches that build an empty DataFrame with the correct column schema when 0 rows are present, matching the CategoricalEncoder fix. SimpleImputer.transform() raises on 0-sample input (ensure_min_samples=1), so the 0-row case must be handled explicitly. Also fix the same .empty pattern in fit() for consistency. Add test_transform_zero_rows_preserves_columns and test_transform_zero_rows_numerical_only_preserves_columns regression tests.
…ility Initialize self._fallback_values in __init__ rather than fit() so the attribute is declared upfront, consistent with the other fit-time attributes and sklearn's estimator cloning expectations.
cgueck895
previously approved these changes
Sep 15, 2026
Contributor
|
The issu with the Categorical Encoder occured when he receives an empty data frame. With the update in de ffill_imputer , this one doesn't output an empty dataframe anymore, unless the original dataframe is empty. In that case, we should check if the datafram for inference is empty in a earlier stage, at the beginning of the inference. Then, the check in the categorcial encoder becomes redundant, but maybe it is also good to have redundancy. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two interacting bugs caused a
ValueError: Length mismatchduring inference when the inference data has fewer columns than the training data (columns filled with NaN being passed toFaultDetector.predict()).Bug 1:
CategoricalEncoder.transform()skips one-hot encoding on 0-row DataFramespd.DataFrame.emptyreturnsTruefor a 0-row DataFrame even when columns exist. Thetransform()method usednot categorical_data.emptyto decide whether to one-hot encode, so on 0-row input it skipped one-hot encoding and returned only the numerical columns. sklearn's_set_outputwrapper then tried to assign the fullget_feature_names_out()(e.g. 21 names) to the narrower DataFrame (e.g. 19 columns), raisingValueError: Length mismatch.Fix: Replaced the
.emptycheck with an explicitself.categorical_columns and len > 0guard and added a branch that returns an empty DataFrame with the correct one-hot encoded column schema when 0 rows are present.Bug 2:
ForwardFillImputer.transform()drops ALL rows when a column is entirely NaNDuring
fit, all-NaN columns are dropped from the tracked column lists. Duringtransform, no symmetric handling existed: if a tracked column became entirely NaN at inference (e.g. a column missing from inference data), forward-fill could not fill it anddropna(how="any")dropped every row — producing the 0-row DataFrame that triggered Bug 1.Fix: During
fit, store per-column training statistics (mean for numerical, most-frequent for categorical) via the packageImputer. Duringtransform, after forward-fill, columns still entirely NaN are filled with their training statistic, so only genuinely-unfillable rows are dropped — not every row.Tests
New regression tests were added covering both bugs, including the numerical and categorical fallback paths. All tests were verified to fail on the unpatched code and pass with the fixes.