Skip to content

fix(quantization): support read-only KV cache fetches in KV cache quantization - #186

Open
vineet-g wants to merge 1 commit into
apple:mainfrom
vineet-g:u/vineetgarg/shared-kv-cache-read-only-quant
Open

vineet-g wants to merge 1 commit into
apple:mainfrom
vineet-g:u/vineetgarg/shared-kv-cache-read-only-quant

Conversation

@vineet-g

@vineet-g vineet-g commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Summary

Finalizing KV cache quantization failed for models whose layers share a KV cache.

Changes

_move_cache_dequant_to_output now treats a cache op with an empty update as a
read-only fetch:

  • Its output is dequantized with the params of the op writing the same slot. A writer and its readers are matched by their non-tensor args (e.g. layer_idx).
  • Its empty update is rebuilt directly in the quantized dtype, so it matches the quantized cache. Casting or reshaping a zero-size quantized tensor fails to lower. Dynamic dims are read from the original update via sym_size.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants