Conversation
After prefill, HyenaCascade.forward routes to sequential_forward, which silently truncated any multi-token input to its last token (u = u[:, -1]): the dropped tokens had zero effect on output and state, the surviving position was computed with a wrong conv/recurrent window, and the layer state advanced by one token instead of seqlen -- all without any error or warning, corrupting every subsequent decode step. Chunk forward with initial state (what e.g. speculative-decoding verification needs) is not implemented on this path, so fail loudly instead of returning silently wrong results. First-party flows are unaffected: Generator feeds one token per step (generation.py) and prefill goes through parallel_forward. Fixes Zymrael#81
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #81.
What
HyenaCascade.sequential_forwardnow raises aValueErroron multi-token input (seqlen > 1) instead of silently truncating tou = u[:, -1].Why
After prefill,
HyenaCascade.forwardroutes every call tosequential_forward. Passing several tokens at once (e.g. a chunk/block forward with initial state, as speculative-decoding verification requires) used to silently discard all but the last token: the dropped tokens had zero effect on output and state, the surviving position was computed with a wrong conv/recurrent window, and the layer state advanced by one token instead ofseqlen— with no error and no warning, corrupting all subsequent decode steps. #81 has a self-contained random-weight reproduction (CPU, ~1 s).A correct chunk forward is a real feature; until it exists, failing loudly turns a silent correctness bug into an explicit, debuggable error.
Why this is safe
No first-party flow can trigger the raise: vortex's own
Generatorfeeds one token per decode step (x = x[:, -1:],vortex/model/generation.pyL169) and prefill goes throughparallel_forward. The new branch only activates for seqlen > 1, which previously always produced wrong results.Test
u = u[:, -1]) — wrong outputs, corrupted state, no error or warning #81 (stepwise single-token reference + one 4-token call) now raises with a clear message at the multi-token call; the single-token stepping path runs unchanged.