A Coding Implementation to Compress and Benchmark Instruction-Tuned LLMs with FP8, GPTQ, and SmoothQuant Quantization using llmcompressor

A tutorial shows how to compress an instruction-tuned LLM with llmcompressor, starting from a baseline FP16 model.

It benchmarks FP8 dynamic quantization, GPTQ W4A16, and SmoothQuant GPTQ W8A8 on size, latency, throughput, and perplexity.

The evaluation also checks sample output quality to compare efficiency gains against accuracy loss.

The results help identify the best quantization approach for deploying on limited hardware.