fix(workflow-operator, v1.2): MAX aggregation checks the wrong empty-group sentinel - #7985
Draft
github-actions[bot] wants to merge 1 commit into
Draft
Conversation
…sentinel (#7532) ### What changes were proposed in this PR? One-line fix in `AggregationOperation.scala`: the `max` aggregation starts its running maximum from the type's minimum value, so its final "was this group empty?" check must compare against that same minimum value, but it compared against the type's maximum value instead (the line was copied from `min`, where that comparison is correct because `min` starts from the maximum). Because the check looked at the wrong sentinel, two results were silently wrong while the workflow completed with no error: - A group with only null values returned the sentinel itself instead of `null`, e.g. `-2147483648` for INTEGER or `1970-01-01 00:00:00` for TIMESTAMP. - A true maximum equal to the type's maximum value was mistaken for "no value seen" and discarded, so `max` over `{1, 5, 2147483647}` reported `5` (the largest value from the other local aggregation worker) instead of `2147483647`. #### Before-and-after Test data: group `g1` has only nulls, `g2` contains `{1, 5, 2147483647}`, `g3` contains `{10, 42}`. <img width="1347" height="900" alt="Screenshot 2026-08-07 at 2 27 40 PM" src="https://github.com/user-attachments/assets/faac65fd-e83f-4952-a0dd-b8109d56334d" /> Before the fix, `max(v)` grouped by `k` returns `-2147483648` for `g1` (expected `null`) and `5` for `g2` (expected `2147483647`): <img width="1341" height="866" alt="Screenshot 2026-08-07 at 2 27 51 PM" src="https://github.com/user-attachments/assets/fdf7f045-aa34-44eb-adee-6e722839d7eb" /> `min(v)` on the same data is correct (`null` for `g1`, `1` for `g2`, `10` for `g3`), confirming only `max`'s empty-group check is broken: <img width="1342" height="868" alt="Screenshot 2026-08-07 at 2 28 00 PM" src="https://github.com/user-attachments/assets/3df9c390-626f-4f01-9ca6-7e634927386b" /> After the fix, `max(v)` returns `null` for `g1`, `2147483647` for `g2`, and `42` for `g3`: <img width="1131" height="756" alt="Screenshot 2026-08-07 at 5 54 17 PM" src="https://github.com/user-attachments/assets/68ba757c-8b4b-4682-b04e-8db740d1583c" /> ### Any related issues, documentation, discussions? Closes #7531 The regression was introduced by #1840, which generalised the hard-coded `Double` sentinels to per-type `minValue`/`maxValue` helpers and updated `maxAgg`'s initialiser but not its finaliser. ### How was this PR tested? Added 5 regression tests, all of which fail without the one-line fix and pass with it (verified in both directions): - `AggregateOpSpec`: `max` over empty input and over all-null input returns `null`; `max` keeps a true maximum equal to the type's maximum value, covering INTEGER, LONG, DOUBLE, and TIMESTAMP. - `AggregationOperationSpec`: a worker-to-final pipeline via `getFinal` keeps `Int.MaxValue` when partial results are re-aggregated, reproducing the two-worker scenario shown above; `max`'s merge stays neutral when one side saw no values. Full aggregate suite: `sbt "WorkflowOperator/testOnly org.apache.texera.amber.operator.aggregate.*"` — 68 tests, all passing. Also verified end to end in the UI with the workflow shown above (screenshots are from before and after rebuilding the backend with this fix). ### Was this PR authored or co-authored using generative AI tooling? Co-authored by: Claude Code (Claude Fable 5) (backported from commit b0dd3ff)
Contributor
Author
|
The cherry-pick conflicted and was committed with conflict markers. Resolve the conflicts on this branch, then mark this PR ready for review. Conflicting files:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this PR?
Automated backport of #7532 to
release/v1.2.Source: b0dd3ff · automation run
Any related issues, documentation, discussions?
Backport of #7532. Originally linked #7531.
How was this PR tested?
Release-branch CI runs on this branch once the conflicts are resolved and this PR is marked ready for review.
Was this PR authored or co-authored using generative AI tooling?
No.