Chapter 3: The Tokenizer - Text to Numbers and Back

This chapter builds a character-level tokenizer that maps text to integer IDs.

It creates a vocabulary from training documents, assigns unique IDs, and adds a BOS token for sequence boundaries.

The tokenizer supports encoding and decoding so text and integers can round-trip correctly.

In MicroGPT’s Chapter3Exercise and Program.cs, input.txt is loaded and the mapping is verified with the sample name “emma.”