Skip to content

[SPARK-58465][PS] Use native expressions for NumPy reciprocal ufunc - #57668

Draft
zhengruifeng wants to merge 11 commits into
apache:masterfrom
zhengruifeng:pandas-native-reciprocal-dev2
Draft

[SPARK-58465][PS] Use native expressions for NumPy reciprocal ufunc#57668
zhengruifeng wants to merge 11 commits into
apache:masterfrom
zhengruifeng:pandas-native-reciprocal-dev2

Conversation

@zhengruifeng

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Replace the scalar pandas UDF mapping for NumPy reciprocal on pandas-on-Spark objects with native Spark SQL expressions.

The expression uses typeof to distinguish Boolean, integral, and floating-point inputs. It preserves NumPy integer reciprocal behavior, including integer-zero overflow, as well as floating-point NaN, infinity, and signed-zero behavior. Add parity coverage for all of these cases.

Why are the changes needed?

The mapping can be evaluated with native Spark SQL expressions, avoiding the Python worker boundary while retaining NumPy-compatible results.

Does this PR introduce any user-facing change?

No. It preserves the existing NumPy-compatible result values and double output type.

How was this patch tested?

  • Added pandas-on-Spark parity coverage for NumPy reciprocal on Boolean, int64, and float64 inputs, including zero, signed zero, NaN, and infinity.
  • Ran JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 SPARK_TESTING=1 SPARK_PREPEND_CLASSES=1 python/run-tests --testnames pyspark.pandas.tests.test_numpy_compat.
  • Ran git diff --check.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Codex (GPT-5)

@zhengruifeng
zhengruifeng marked this pull request as draft July 31, 2026 09:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant