Make stream deletion asynchronous, resumable across restarts

[prabhaks]

Summary

Stacked on #1768.

Closes the second half of #1763: deleting a large stream currently blocks the DELETE request on a full recursive object-store delete, which can take a long time for TB-scale streams even though the underlying delete itself is already reasonably efficient (batched, concurrent). This PR moves the actual deletion to the background, building on the tombstone/deleting-flag safety net added in #1768.

• DELETE /logstream/{stream} now writes a durable tombstone, best-effort deletes the small stream.json so the stream disappears from listings almost immediately, flags the stream deleting in memory, fans the delete out to ingestors, and responds 202 Accepted instead of 200 OK once all of that is durably in place -- before the slow part even starts.
• The actual recursive object-store delete runs in a background task (deduplicated per stream), clearing the tombstone and removing the stream from memory once it finishes.
• If the node crashes or restarts mid-deletion, the tombstone is discovered on startup and the deletion resumes automatically -- no manual cleanup needed.
• Only the node that received the original client request ever runs the physical delete. Ingestors just flag the stream as deleting and wait for the tombstone to clear, so a single deletion isn't redundantly re-run by every node in the cluster.
• A periodic self-heal check catches a node that missed the live notification (e.g. it was down or partitioned at the time) and brings it back in sync within one sync interval.
• Fixed a bug (found during review of this change) where list_streams() on the local filesystem backend would fail the entire listing if it encountered a stream mid-deletion, since that backend treats a stream directory without stream.json as corrupt rather than "not a stream."

API contract change

DELETE /logstream/{stream} now returns 202 Accepted (body: "log stream {name} deletion started") instead of 200 OK once the deletion has finished. Any client code checking for exactly 200 will need updating.

Test plan

• cargo build --lib
• cargo test --lib (449 passed)
• cargo fmt --check
• cargo clippy --lib --all-targets
• New unit tests: ACTIVE_STREAM_DELETIONS dedup semantics, and list_streams() correctly skipping (not erroring on) a stream mid-deletion on the local filesystem backend, including a control case confirming a genuinely corrupt directory still errors
• Live cluster validation (202 timing, crash-mid-deletion resume, ingestor self-heal) -- to be run separately against a real multi-node cluster

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}

[prabhaks]

CI status update: the two Quest integration test failures (Distributed and Standalone) are an expected consequence of this PR's intentional API contract change, not a bug in the implementation.

Root causes, confirmed from the CI logs:

• DELETE /logstream/{stream} now returns 202 Accepted instead of 200 OK (by design, since the deletion is now asynchronous). Every Quest test whose setup/teardown does "delete stream, assert 200" fails at that assertion and then runs in a partially-cleaned-up state, which cascades into several unrelated-looking failures later in the same sequential test run.
• Recreating a stream immediately after deleting it can now correctly return 409 Conflict ("being deleted, please retry shortly") instead of silently succeeding, since the old stream may still be mid-deletion. A few tests that delete-then-immediately-recreate a stream with the same name hit this.

Both are already called out under "API contract changes" in the PR description. Quest (quay.io/parseablehq/quest:main) is a separate repo/image and will need its assertions updated to expect 202 for stream deletion and to tolerate/retry on a transient 409 when recreating a stream right after deleting it, before this PR's CI can go green.

[prabhaks]

Opened a companion fix for the Quest test suite: parseablehq/quest#126 (updates the hardcoded 200 assertions to 202, and adds retry-on-409 for tests that recreate a stream right after deleting it).