Google introduces TurboQuant, cutting LLM memory usage by 6x with no accuracy loss

Google developed three AI compression algorithms, google developed three compression algorithms named TurboQuant, PolarQuant, and Quantized Johnson-Lindenstrauss.

TurboQuant family use vector quantization to reduce memory footprint the algorithms use vector quantization to reduce model memory footprint.

TurboQuant reduces key-value cache memory footprint by at least 6×, turboQuant reduces the key-value cache memory footprint by at least 6× while maintaining performance.

Uses PolarQuant to randomly rotate vectors prior to quantization, turboQuant uses PolarQuant to rotate data vectors before applying a high-quality quantizer.

Applies 1-bit Quantized Johnson-Lindenstrauss for error correction, turboQuant applies a 1-bit Quantized Johnson-Lindenstrauss step as an error-correction mechanism.