Understand and Accelerate Memory Processing Pipeline for Disaggregated LLM Inference

The paper defines a four-step memory processing pipeline for long-context LLM inference: Prepare Memory, Compute Relevancy, Retrieval, and Apply to Inference.

Profiling shows memory processing can account for 22%–97% of LLM inference time and has heterogeneous computational traits.

On a GPU+FPGA system (AMD MI210 + Alveo U55C), offloading memory-bound tasks to the FPGA yielded 1.04–2.2× speedups and 1.11–4.7× energy savings versus a GPU baseline.