Paged Attention in Large Language Models LLMs

Model configuration uses 524,288 bytes per token for KV storage, gPT-style config with 32 layers, 32 heads, 128 head dim, fp16 storage.

Naive allocator pre-allocates memory for 2048 tokens per request, mAX_SEQ_LEN set to 2048 tokens.

Request usage actually uses memory for 500 tokens on average, aVG_RESPONSE set to 500 tokens.

Naive allocator pre-allocation results in about 1024 MB reserved per request calculation based on KV_BYTES_PER_TOKEN and MAX_SEQ_LEN.

Request usage actual memory written per request is about 250 MB based on AVG_RESPONSE and KV_BYTES_PER_TOKEN.