Memory-Bound but Not Bandwidth-Limited: The Physical AI Inference Gap in Batch-1 LLM Decode

A benchmark of batch-1 LLM decoding finds latency is limited by more than memory bandwidth alone.

Faster GPUs like H100 do not always sustain proportionally higher achieved bandwidth, and CUDA Graphs cuts latency far more on H100 than on L4.

Several quantized inference paths fail to realize the expected weight-traffic savings, with GPTQ plus ExLlamaV2 the main exception.

The results help explain why single-stream inference for robots and other physical AI systems can stay slow even on strong hardware.