Skip to content

[AMORO-4348] Fix infinite retry loop for failed table processes - #4349

Merged
xxubai merged 2 commits into
apache:masterfrom
j1wonpark:fix/process-infinite-retry-loop
Sep 2, 2026
Merged

[AMORO-4348] Fix infinite retry loop for failed table processes#4349
xxubai merged 2 commits into
apache:masterfrom
j1wonpark:fix/process-infinite-retry-loop

Conversation

@j1wonpark

Copy link
Copy Markdown
Contributor

Why are the changes needed?

Fix #4348.

Since #4116, ProcessService constructs DefaultTableProcessStore passing the current retry count into the constructor's maxRetryTime parameter. A new process therefore gets maxRetryTime = 0 and becomes terminal on its first failure, every subsequent transition is rejected, and retryNumber never increases — while the retry branch in executeOrTraceProcess keeps evaluating retryNumber < PROCESS_MAX_RETRY_NUMBER as true and resubmits the process forever, with no backoff.

Brief change log

How was this patch tested?

  • Add some test cases that check the changes thoroughly including negative and positive cases if possible

TestDefaultProcessService#testFailedProcessRetryIsBounded reproduces the infinite loop before the fix (the process is never dropped and the wait times out) and passes after: the failing process is submitted 1 + PROCESS_MAX_RETRY_NUMBER times in total and then untracked. Existing process service tests all pass.

  • Run test locally before making a pull request

Documentation

  • Does this pull request introduce a new feature? (no)

Signed-off-by: Jiwon Park <jiwonpark@apache.org>
@github-actions github-actions Bot added the module:ams-server Ams server module label Aug 31, 2026
@xxubai xxubai added this to the Release 0.9.0 milestone Sep 1, 2026

@Override
public ProcessStatus getStatus(String processIdentifier) {
if (failedIdentifiers.contains(processIdentifier)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Please make the fake engine fail directly from submitTableProcess().

Returning FAILED immediately takes the fast-terminal path and produces redundant COMPLETE_FAILED transition errors.

Suggested change
if (failedIdentifiers.contains(processIdentifier)) {
throw new IllegalStateException("Submission failure");

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@zhoujinsong zhoujinsong left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Signed-off-by: Jiwon Park <jiwonpark@apache.org>
@xxubai
xxubai merged commit 05c0efe into apache:master Sep 2, 2026
7 checks passed
@j1wonpark
j1wonpark deleted the fix/process-infinite-retry-loop branch September 2, 2026 22:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

module:ams-server Ams server module

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Failed table process is resubmitted in an infinite retry loop

3 participants