Skip to content

perf(desktop): coalesce streamed text chunks before transport emit - #2164

Open
YodonTan wants to merge 2 commits into
GCWing:mainfrom
YodonTan:perf/coalesce-text-chunk-events
Open

perf(desktop): coalesce streamed text chunks before transport emit#2164
YodonTan wants to merge 2 commits into
GCWing:mainfrom
YodonTan:perf/coalesce-text-chunk-events

Conversation

@YodonTan

@YodonTan YodonTan commented Aug 7, 2026

Copy link
Copy Markdown

Why

During streaming AI responses, the agent runtime emits one TextChunk / ThinkingChunk event per provider chunk. The desktop event loop currently forwards every chunk to the WebView as a separate Tauri IPC message (JSON serialization, WebView2 boundary crossing, JS parse + dispatch) and, when peer devices are attached, as a separate end-to-end encrypted relay message.

This change coalesces streamed text/thinking chunks in the desktop event loop before the transport emit:

  • UI performance: fewer IPC messages mean fewer JS event dispatches, and the frontend's 32ms EventBatcher becomes arrival-driven for fast streams (~20Hz instead of ~31Hz store/React updates), reducing re-renders and layout churn while streaming.
  • Remote / network savings: each peer-fanout event currently costs serialize + encrypt + base64 + relay + decrypt/parse on the mobile side; coalescing cuts the message rate by roughly 3-10x, proportionally reducing encrypted relay traffic and mobile wakeups.
  • IPC resources: fewer messages crossing the WebView2 boundary (per-message JSON serialization and parsing is amortized away).

How

  • Merge chunks sharing the same (session, turn, round, attempt, contentType) within a 50ms time window; text is concatenated and the thinking isEnd flag is OR-ed.
  • Non-chunk events (completion, error, cancellation, tool events) flush pending merged text first and then pass through, preserving per-stream ordering: text always precedes completion/error/cancellation.
  • Flush order preserves the producer's FIFO sequence (first-arrival per stream), so merged thinking precedes merged text for the same stream.
  • The window is a throttle, not a debounce: the deadline is armed once and never extended by a steady chunk stream; it is cleared once a flush drains the buffer.
  • Internal event subscribers (event_router.route) still receive the raw per-chunk events; only WebView / peer delivery is coalesced.
  • Content stays byte-equivalent to what the frontend already accumulates (its EventBatcher merges by the same key), ordering is preserved, and ACP/TUI/CLI consumers plus event contracts are untouched. Text delivery adds at most ~50ms, absorbed by the frontend typewriter.

Fixes #2161

@GCWing
GCWing requested a review from wsp1911 August 8, 2026 01:04

@wsp1911 wsp1911 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] 避免队列排空循环饿死 50ms flush timer

src/apps/desktop/src/lib.rs:2244tokio::select! 只会在进入队列分支前 poll 一次 flush timer。一旦选中队列分支,2246-2265 的内层循环就会持续调用 dequeue_configured_batch(),直到取到空 batch,期间不会回到外层 select!

这会产生两个相关问题:

  1. 如果生产者持续让队列保持非空,timer 就不会被再次 poll,已经到期的 deadline 也无法触发 flush。
  2. 更关键的是,首次 deadline 直到排空循环结束后,才在 2272-2277 设置。因此首个 chunk 进入 coalescer 时,实际上并没有立即开始它的 50ms 窗口。

在持续流式输出下,如果入队速度不低于 event_router.route()deliver_event_to_webview() 的处理速度,队列可能长期保持非空。此时已缓冲的文本可能等待远超 50ms,直到队列出现空隙,或 non-chunk 事件通过 coalescer.push(non_chunk) 强制 flush。这违背了 PR 所描述的 throttle 语义,也无法满足“文本交付最多增加约 50ms 延迟”的保证。

建议限制每次队列分支的处理量,例如处理一个 batch 或有限数量事件后就回到 select!,并在首个 chunk 被缓冲时立即设置 deadline。另一种方式是在 drain 循环中检查 flush deadline,到期后先执行 flush,再继续处理队列。最好再补充一个“队列持续非空”的回归测试,覆盖真实 event loop 的调度行为。

@YodonTan

YodonTan commented Aug 9, 2026

Copy link
Copy Markdown
Author

Thanks for the review. The P1 issue is fixed in 9c2d69724:

  1. Deadline armed at the first buffered chunk — the window is armed per envelope inside the drain loop, so it counts from the first chunk instead of from the end of the drain.
  2. Expired deadline honored inside the drain — the queue branch checks Instant::now() >= deadline after each batch and flushes in place before processing continues, so text waits at most one window even when the queue never empties (the window timer is only polled at the outer select!).
  3. Regression tests for the sustained-non-empty-queue schedule (paused clock, deterministic): flush_timer_fires_while_queue_stays_non_empty asserts the first flush lands ~one window after the first chunk and further windows keep firing under load; sustained_drain_does_not_lose_or_duplicate_text asserts the merged chunk sequence stays contiguous. Coalescer unit tests: 18 passed; driver scheduling tests: 2 passed.

The same commit also adds the rate-adaptive window (30-100ms, calibrated at the measured median stream rate, 50ms at average speed) so fast streams merge more chunks per message while slow streams keep a small window, plus a fix to count buffered content in Unicode characters instead of UTF-8 bytes (Chinese text would otherwise overestimate the stream rate 3x).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Optimize]: coalesce streamed text chunks before Tauri emit to reduce IPC and peer fanout pressure

2 participants