Make delete-by-query conflict retry tests deterministic and guarantee writer cleanup

[niemyjski]

Observed during the production-readiness audit of #314

The full Release build succeeded (zero warnings/errors), but this run failed one existing test:

https://github.com/FoundatioFx/Foundatio.Repositories/actions/runs/35742581329/job/106795844454

• Commit: 4fd1a96411277d2a2865f94c2cd3580fa4e9158e
• Test: RepositoryTests.RemoveAllAsync_DeleteByQuery_AccumulatesDeletedCountAcrossConflictRetries
• Expected deleted count: 1000; actual: 998.
• Log: RemoveAll: 998 records removed after 11 attempts (2 unresolved version conflicts).
• Suite result: 843 total, 842 passed, 1 failed, 0 skipped.

A subsequent full run passed after adding the independent multi-get cache regressions; the delete-by-query implementation and these tests were unchanged:
https://github.com/FoundatioFx/Foundatio.Repositories/actions/runs/35745192085

Why the current fixture is not deterministic

The tests introduced with #291 start StartConflictWriterAsync, await RemoveAllAsync, and only then cancel the writer. The writer repeatedly patches surviving documents until cancellation. Contrary to the test comment, catching DocumentNotFoundException only logs and continues; it does not stop the writer. A bounded delete retry loop therefore cannot guarantee complete deletion while that writer remains active.

The one-pass warning test also uses Task.Delay(100) to claim a guaranteed conflict. Elapsed time does not establish that a write occurred between the delete-by-query snapshot and its delete operation.

Writer shutdown is not protected by finally: if the operation under test throws, the writer may outlive the test and interfere with subsequent shared-index tests until its timeout.

This is a test-contract/scheduling problem, not evidence that the production retry budget should be increased or made unbounded. Returning a partial count and warning after exhausted retries is the documented contract.

Required repair

1. Replace timing-dependent conflict generation with a controlled request/response sequence or an explicit synchronization seam. Prove that the first attempt has a conflict before allowing the writer to stop; make the successful-retry case quiescent before the succeeding attempt. A scripted transport fixture is appropriate for retry accounting, retry-budget, and warning assertions; retain a real-Elasticsearch integration case for ordinary deletion.
2. Assert cumulative deletion across multiple actual attempts, exact retry-budget behavior, and one-time notification side effects. Do not weaken the expected count, skip tests, or rely on larger delays/timeouts/retry counts.
3. Make persistent-conflict tests deterministic independently of the successful-convergence tests.
4. Always cancel and await any writer in finally, including failure/cancellation paths.

Verification

Run the repository's CI command with the documented Elasticsearch services running:

dotnet test --solution ./Foundatio.Repositories.slnx --configuration Release --no-build --report-github

Before that command, build the same Release solution. Repeat the retry tests/full CI runs to check stability, and demonstrate that deliberately broken count accumulation or retry-budget handling causes the deterministic tests to fail. Confirm no background writer remains after a failed test.

[niemyjski]

The final #314 audit is replacing the four scheduling-dependent conflict tests rather than increasing delays/retry limits or weakening their assertions. The replacement removes both conflict-writer helpers entirely.

The prepared coverage has two layers:

• A scripted IRequestInvoker exercises the real Elasticsearch response serializer and repository retry loop, with exact request budgets, cumulative deletion counts, exhausted-budget warnings, once-per-operation notifications, disabled notifications, and request failures.
• A real-Elasticsearch fixture disables scheduled refresh, performs explicit initial refresh followed by unrefreshed updates, and first asserts a native delete-by-query probe returns exactly three version conflicts with zero deletions. It then checks Retry(0) returns seven deletions with three documents remaining, while Retry(1) returns ten cumulative deletions with none remaining. Index settings are restored in finally; there is no writer task or timing delay.

Validation is isolated from the PR branch in https://github.com/FoundatioFx/Foundatio.Repositories/actions/runs/35767909698. It includes repeated executions, deliberately broken accumulation/budget mutations that must fail the intended assertions, and the full solution suite. This comment records the repair design, not a passing result: the final validation job is still queued and the changes have not yet been promoted to #314.

[niemyjski]

The deterministic repair is now delivered on PR #314 (head b92b5c8f0dc1d28e1827c4dde94be5ee5fa3ee21), not just staged on an audit branch.

• 62ceb3e52e6d9f79e3903413ebb26507ba66a8cc removes the four timing-dependent conflict tests and both writer helpers, replacing them with ten scripted-transport cases. The ordinary live-server deletion test is retained.
• 27b923584418e0a5e905c4d12d26012ccf084355 adds two real-Elasticsearch cases with scheduled refresh disabled and known unrefreshed updates. A probe must report exactly three version conflicts and zero deletions before the operation under test. Retry(0) then returns seven deletions with three remaining; Retry(1) returns ten with none remaining. Settings are restored in finally with a bounded cleanup token.
• b92b5c8 makes the scripted transport honor Elasticsearch product headers and synchronous mapping requests, and rejects unexpected fixture errors. Production retry logic and budgets are unchanged.

Verified in https://github.com/FoundatioFx/Foundatio.Repositories/actions/runs/35779319164 on the exact delivered commit:

• Full Release solution suite: 887 passed, 0 failed, 0 skipped (728 Elasticsearch-project cases plus 159 core-project cases).
• Twenty consecutive runs of the 27 scripted retry/MGET cases: 540 passed.
• Five consecutive runs of both real-conflict cases: 10 passed.
• Deliberately replacing cumulative addition with assignment produced six failing tests; deliberately limiting the retry budget to one attempt produced seven failing tests. Both included the intended count/notification regression. Original source was restored and rebuilt before full-suite validation.
• Changed-file formatting, documentation build, and creation of both library packages passed.

The evidence artifact records validated commit b92b5c8f0dc1d28e1827c4dde94be5ee5fa3ee21 and included main a61d541613528428cc00bc2236602db72948a924. PR #314 contains Fixes #328, so this issue remains open until the repair is merged.