KurrentDB ReadEvents buffers the entire requested range before yielding — IAsyncEnumerable is not actually streaming

[alexeyzimarev]

Summary

KurrentDBEventStore.ReadEvents (and ReadEventsBackwards) return IAsyncEnumerable<StreamEvent>, but the implementation materializes the ENTIRE requested range before the first yield — verified at current HEAD (3cb68c2, and identical in 0.16.5-alpha.0.7 which we pin):

// src/KurrentDB/src/Eventuous.KurrentDB/KurrentDBEventStore.cs
public async IAsyncEnumerable<StreamEvent> ReadEvents(StreamName stream, StreamReadPosition start, int count, ...) {
    var read = _client.ReadStreamAsync(Direction.Forwards, stream, start.AsStreamPosition(), count, ...);

    var events = await TryExecute(
        async () => {
            var resolvedEvents = await read.ToArrayAsync(cancellationToken).NoContext();   // <— full ResolvedEvent[]
            return ToStreamEvents(resolvedEvents);                                          // <— second full StreamEvent[]
        },
        ...
    );

    foreach (var evt in events) yield return evt;   // yields only after both arrays exist
}

So a read holds two stream-sized arrays (raw ResolvedEvent[] + fully-deserialized StreamEvent[], with per-event payload copies) regardless of how the caller consumes the enumerable. The IAsyncEnumerable signature strongly implies streaming; callers naturally write await foreach folds expecting O(1)/O(batch) memory and silently get O(stream).

Real-world impact

We run Kurrent Capacitor (records AI coding-agent sessions; one stream per session, routinely 10-20k+ events with fat text payloads) on small containers (1 GiB / 768 MB heap ceiling). Several code paths did the natural thing — ReadEvents(stream, Start, int.MaxValue, ct) as "read to end", some even via await foreach folds that never retain the events — and a single large stream OOM'd the pod in production: the transient double-buffer alone was the spike. There is no explicit read-to-end API, so count: int.MaxValue is the idiom users reach for, and it turns every large stream into a heap bomb.

We've since worked around it downstream with a paging extension (bounded per-call count, advancing from the last page's revision), but that compensates for the adapter rather than fixing it — and any other Eventuous user making the same reasonable assumption about the IAsyncEnumerable contract is exposed the same way.

What we actually need

1. True streaming ReadEvents: yield each event as the underlying ReadStreamAsync produces it — no intermediate arrays. The exception-mapping currently done via TryExecute (the reason for ToArrayAsync, presumably) can move inside the iterator: wrap MoveNextAsync in the try/catch that maps StreamNotFoundException/errors, and yield return outside the try:

   await using var e = read.GetAsyncEnumerator(cancellationToken);
   while (true) {
       bool moved;
       try { moved = await e.MoveNextAsync().NoContext(); }
       catch (StreamNotFoundException) { throw new StreamNotFound(stream); }
       catch (Exception ex) when (ex is not OperationCanceledException) { throw new ReadFromStreamException(stream, ex); }
       if (!moved) yield break;
       yield return ToStreamEvent(e.Current);
   }

   Same shape for ReadEventsBackwards.

2. A first-class read-to-end (e.g. an overload without count, or a documented ReadStreamToEnd), so int.MaxValue stops being the idiom.

3. Documented memory semantics on IEventReader.ReadEvents — the interface contract should say whether implementations stream or buffer, so adapter authors and consumers stop guessing.

(1) alone solves the problem; (2)/(3) prevent the next user from rediscovering it.

Happy to contribute a PR for the streaming change if you're open to it.

🤖 Generated with Claude Code

[linear]

EVE-242