A Coding Tutorial on OpenMythos on Recurrent-Depth Transformers with Depth Extrapolation, Adaptive Computation, and Mixture-of-Experts Routing

The tutorial builds and evaluates OpenMythos, a Claude Mythos-style recurrent-depth transformer, using GQA and MLA variants.

It compares KV-cache memory use and recurrent stability, then trains the model on a parity benchmark with Adam.

At inference time, it studies depth scaling, ACT halting, and Mixture-of-Experts routing to probe adaptive computation.

The takeaway: iterative depth can improve reasoning and efficiency without growing model size.