We got 207 tok/s with Qwen3.5-27B on an RTX 3090 | Hacker News
TL;DR AI
2 min readKey summary
A standalone C++/ggml inference stack for Qwen3.5-27B hit up to 207.6 tok/s on a single RTX 3090.
The setup uses speculative decoding with DFlash and DDTree, outperforming llama.cpp and other inference stacks in throughput.
KV quantization and memory optimizations let the model fit a 128K context window into 24 GB of VRAM.
The result shows that a 27B hybrid model can run much faster on consumer hardware with custom decoding and compression techniques.
Same topic
