Conversation
DecodeServiceOutbox reports incremental snapshot text. The decode-service client throttled live updates but dropped the text it skipped: once the first visible chunk was yielded, every later .token event carried an empty textDelta. Consumers that accumulate .token events therefore saw only the first chunk, while generation itself completed normally (finish_reason "stop", full completion_tokens). AppModel hides this because it rebuilds the answer from GenerationTranscriptMailbox. AppServerInferenceBackend, which serves /v1/chat/completions, does not, so the endpoint returned a single word for a full answer. Carry the pending text forward and flush it before the terminal event.
|
I have made a patch which is Swift 6.4/Xcode 27 compatible. Meaning that it works. I am not a developer so i need to Trust Claude here a bit much |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes loss of incremental text deltas in
DecodeServiceInferenceClientwhen snapshot events are throttled.Root cause
DecodeServiceOutboxemits incrementaltextDeltavalues. During snapshot throttling, the previous implementation discarded deltas that arrived between emitted snapshots, causing the OpenAI-compatible server path to receive only the first visible chunk even though inference completed normally.Fix
Accumulate throttled
textDeltavalues inpendingVisibleText, emit the accumulated text when a snapshot is forwarded, and flush any remaining text before the terminal event.Verification
With the real model and inference path:
ManThe generated token count remains unchanged; the fix prevents intermediate text from being dropped.
Scope
This PR contains only the production fix in
DecodeServiceInferenceClient.swift.Unrelated Swift 6.4/Xcode 27 compatibility changes are intentionally excluded.
Testing limitation
The repository does not currently compile cleanly with Swift 6.4 / Xcode 27 because of pre-existing region-isolation and actor-isolation diagnostics in unrelated files. Behavioral verification above was obtained with local-only compatibility workarounds plus a temporary diagnostic probe; neither is part of this PR. The fix itself is a one-file change with no new dependencies or public API.