Perplexity AI Open-Sources Unigram Tokenizer That Achieves 5x Lower p50 Latency Than Hugging Face tokenizers Crate

Perplexity AI rebuilt its Unigram tokenizer in Rust and open-sourced it in the pplx-garden repository.

The company says it delivers about 5x lower p50 latency than Hugging Face tokenizers at production lengths.

It also uses less CPU and avoids steady-state heap allocations, making it more efficient for inference pipelines.

The speedup matters because tokenization can be a CPU-side bottleneck for smaller LLM systems and production deployments.