Feature request: Do No Continue Processing Event in Batch Mode for Kinesis/DDBStreams

[hjgraca]

Use case

When processing a DDB Stream in batch mode, I want to stop processing when a failure is reached. Since this is a stream, and ordering of message is important for me, the processing should immediately stop.

That is to say, if my data is partitioned on Purchase ID, I want to ensure all events related to the same purchase are played in order. If a failure occurs, the processing of the stream should stop and retry later.

Initial issue and more information: aws-powertools/powertools-lambda-java#1820

Solution/User Experience

Need to review the Error Handling policy to see if we already support this feature
https://docs.powertools.aws.dev/lambda/dotnet/utilities/batch-processing/#error-handling-policy

Alternative solutions

Acknowledgment

• This feature request meets Powertools for AWS Lambda (.NET) Tenets
• Should this be considered in other Powertools for AWS Lambda languages? i.e. Python, Java, and TypeScript

[lachriz-aws]

I would argue that we already support this feature request using the ErrorHandlingPolicy setting.

If any type of Exception occurs during record processing, we check the ErrorHandlingPolicy to decide what to do. If it equals StopOnFirstBatchItemFailure then we throw a CircuitBreakerException, see:

powertools-lambda-dotnet/libraries/src/AWS.Lambda.Powertools.BatchProcessing/BatchProcessor.cs

Lines 218 to 225 in dfb8ae4

	// Check if we should stop record processing on first error
	if (errorHandlingPolicy == BatchProcessorErrorHandlingPolicy.StopOnFirstBatchItemFailure)
	{
	// This causes the loop's (inner) cancellation token to be cancelled for all operations already scheduled internally
	throw new CircuitBreakerException(
	"Error handling policy is configured to stop processing on first batch item failure. See inner exception for details.",
	ex);
	}

... which causes the outer loop (the one that iterates the batch) to stop iterating further, see:

powertools-lambda-dotnet/libraries/src/AWS.Lambda.Powertools.BatchProcessing/BatchProcessor.cs

Lines 125 to 128 in dfb8ae4

	catch (Exception ex) when (ex is CircuitBreakerException or OperationCanceledException)
	{
	// NOOP
	}

Important note from the docs about when this feature is used together with parallel processing:

When using StopOnFirstBatchItemFailure and parallel processing is enabled, all batch items already scheduled to be processed, will be allowed to complete before the batch processing stops.
Therefore, if order is important, it is recommended to use sequential (non-parallel) processing together with this value.

[hjgraca]

Thank you @lachriz-aws , really appreciate the feedback. I agree and that is my understanding as well.

I need to review with other runtimes to confirm if it covers all use cases.