KV Cache Compression

SIGNAL ANALYSIS

Transformer inference is bottlenecked by key-value cache memory costs, which grow with batch size and context length. A new method combines Tucker and JL-Residual allocation to compress the cache with minimal loss. This could significantly improve throughput for long-context models.