Building the foundation for running extra-large language models

Cloudflare explained how Workers AI is scaling extra-large open-source language models for agentic use cases.

The system uses prefill-decode disaggregation, splitting input-token processing and output-token generation across separate servers.

A token-aware load balancer routes requests and moves KV cache state between stages to keep inference efficient.

Cloudflare says the design improves speed, flexibility, and GPU utilization, and can be tuned for input-heavy or output-heavy traffic.