[mskarayigit]

retry_type variables in the TestClass now check within the retry_prompt in the Sequencer.py

• RT_ABORT : Abort the whole test sequence after all attempts have failed
• RT_RETRY : Automatically retry up to "attempts" and then prompts the user for the options (Retry, Abort, Fail)
• RT_PROMPT : is the default in the TestClass, Prompt the user; Options are Abort the sequence, retry, or fail and continue (currently has the same functionality as RT_RETRY)
• RT_FAIL : Automatically fails and moves on onto the next test if all the number of attempts have failed.

Also, added a test script under test.

[skarayigit]

The attempts variable in the TestClass has been configured to automatically retry a test case when a failure occurs. This functionality is working as intended. However, an additional requirement is needed to allow the overall test sequence to continue executing on the remaining circuits even after all retry attempts for a particular test have been exhausted and the test ultimately fails.

This approach would enable the generation of a comprehensive test report at the end of the execution, identifying all failed circuits without requiring operator intervention or waiting for user input. As a result, any necessary rework can be performed on all failed circuits in a single batch, improving efficiency and avoiding the need to repeatedly repair and retest individual circuits one at a time.

In some situations, test failures may be caused by transient issues such as instrument switching delays or measurement timing variations. In these cases, one or more retry attempts may successfully recover from the issue and result in a passing test outcome.

Conversely, certain test cases may be associated with critical circuits or dependencies that are essential for downstream testing. For these scenarios, it should be possible to configure a maximum number of retry attempts and, if the test continues to fail after all retries, terminate the test sequence. This prevents subsequent tests from being executed when their results would no longer be valid due to the failure of a critical prerequisite circuit.

Therefore, the test framework should support both behaviors:

Non-critical tests (RT_FAIL): Continue testing the remaining circuits after all retry attempts have failed, and record the failure for reporting purposes.
Critical tests(RT_ABORT): Abort the test sequence when all retry attempts have been exhausted and the test still fails, preventing invalid downstream testing.

RT_PROMPT is still a default behavior and prompts the user after all attempts have failed with options (RETRY, ABORT, FAIL)

This flexibility would improve test efficiency, reduce operator involvement, and provide more meaningful test reports and failure analysis.

[jcollins1983]

There are a couple of things to address. Perhaps yourself, Daniel and I can sit down when you're back at CSM to discuss the logic a bit more before going through another cycle on the PR?

[jcollins1983]

Can you move these comments into the doc string? That way instead of

We get

[jcollins1983]

This error message is not really all that helpful.
If the person writing the tests ignores the hints that the IDE will be giving them because of the type hinting, and they pass an int, they will be told that what they passed in must be a valid int.

I think something like the below will be clearer.

            raise TypeError(
                f"Invalid retry_type type: {type(retry_type).__name__}. retry_type must be a valid {TestClass.RetryMode.__name__} member"
            )

[jcollins1983]

These 2 cases are currently identical. If attempts in the test class is > 1, the test will be attempted attempts times regardless of whether we have set RT_PROMPT or RT_RETRY.

I think we have 2 options.

1. Where RT_PROMPT should prompt regardless of the number of attempts that has been set and RT_RETRY is the only setting where a test will automatically retry, and potentially only allow at most attempts retries. This would require a deeper change to the logic.
2. We remove one of them, I vote for RT_RETRY because the (automatic) retry attempts is set by the attempts variable for the test class, and either way we will end up with a prompt for further retries or moving on.

My personal vote is for option 2.

@daniel-montanari thoughts?

[daniel-montanari]

I think the logic should be moved out a level.
Have retry_prompt be the thing that purely handles the prompt. If we have a prompt AND we are non-interactive then the default behaviour is to fail.

Then outside of this function implement the logic for the other RT types and switch off either the prompt or non-interactive RT type

RT_RETRY should return True (though I am not a fan of auto retrying tests, it is probably still a valid thing to do)
RT_FAIL should return False
RT_ABORT should raise SequenceAbort

[daniel-montanari]

Yeah having another look at this the whole loop is broken.
If you use RT_RETRY as I want it to behave above then this would cause an infinite loop that ignores the number of attempts.

I don't understand why abort_exceptions allows retries. self.ABORT inside of run_test also doesn't seem to get used as it only gets set after the test has exited.

I think the attempts and retry logic should be moved out of run_test so the only place to deal with it is in run_once. Also those names seem backwards to me.

[jcollins1983]

You get bonus points for writing tests without being prompted to. But these are manual tests, so should be in the manual folder.
Having said that though, I think that can be incorporated into the sequencer tests test/core/test_sequencer.py and they can all be automated.

[daniel-montanari]

I think we don't strictly need this to achieve the behaviour we want. Using "amptest_non-interactive" as the programming config value should work. The number of attempts will be limited to 1, but I would argue that tests should be reliable to pass first try.