Google Unveils TurboQuant AI Compression Tool

Google Research unveiled TurboQuant, a novel compression algorithm designed to slash memory demands in artificial intelligence systems by at least six times during inference operations, while preserving full model accuracy. The announcement, detailed in a research paper published on Tuesday, prompted an immediate market reaction as shares in memory chip makers declined, reflecting investor concerns over potential shifts in AI hardware needs.

TurboQuant targets the key-value (KV) cache, a critical component in large language models (LLMs) that stores attention data to avoid recomputing it with each new token generated. As models handle longer contexts, such as conversations spanning hundreds of thousands of tokens, the KV cache can swell to consume hundreds of gigabytes of GPU memory per session, limiting scalability and driving up costs. Traditional compression techniques, like quantization, reduce data precision from 32-bit floats to lower-bit integers but often require storing extra constants for decompression, adding one to two bits per value and eroding efficiency gains.

The algorithm overcomes this through a two-stage process. First, PolarQuant transforms vectors from...