Moonshot AI and Tsinghua Researchers Propose PrfaaS: A Cross-Datacenter KVCache Architecture that Rethinks How LLMs are Served at Scale

Moonshot AI and Tsinghua proposed PrfaaS, a disaggregated LLM serving design that separates prefill from decode.

Long-context prefill runs on dedicated compute-heavy clusters, while only the smaller KVCache is sent over commodity Ethernet to local decode clusters.

On an internal 1T-parameter hybrid model, PrfaaS improved throughput by 54% over a homogeneous baseline and by 32% over a naive heterogeneous setup.

The results suggest cross-datacenter LLM serving can scale beyond RDMA limits by leveraging hybrid-attention models with much smaller KVCache.