Meet Gigatoken: A Rust BPE Tokenizer that Encodes Text at 24.53 GB/s, up to 989x Faster than HuggingFace Tokenizers

Gigatoken, an MIT-licensed Rust BPE tokenizer with Python bindings, was released and benchmarked at very high throughput.

Its speed comes from hand-optimized pretokenization using SIMD-style techniques, SWAR, dual-cursor execution, caching, and a native Rust API.

On GPT-2/OpenWebText-style benchmarks, it reports up to 24.53 GB/s and outperforms HuggingFace tokenizers and OpenAI tiktoken on multiple CPUs.

The result suggests tokenizer performance can be improved by orders of magnitude, potentially easing a major bottleneck in LLM training and inference.