This AI Paper Introduces TinyLoRA, A 13-Parameter Fine-Tuning Method That Reaches 91.8 Percent GSM8K on Qwen2.5-7B

TinyLoRA achieved 91.8% accuracy on GSM8K with 13 parameters, tinyLoRA achieved 91.8% accuracy on GSM8K using Qwen2.5-7B-Instruct with 13 trainable parameters (26 bytes in bf16).

Scales down to a single trainable parameter, the TinyLoRA parameterization can scale down to a single trainable parameter under extreme weight-tying settings.

Uses a trainable vector projected via a fixed random tensor for updates, tinyLoRA replaces the trainable matrix with a low-dimensional trainable vector v projected through a fixed random tensor P in the update rule W' = W + UΣ(∑_i v_i P_i)V^T.

Weight tying reduces total trainable parameters via ntie scaling, applying a weight tying factor (ntie) makes total trainable parameters scale as O(nm u/ntie), enabling shared updates across modules and layers.

Standard LoRA has a minimum update size that scales with layer width and rank, standard LoRA parameter count scales with layer width and rank and has a nontrivial lower bound (e.g., ~3 million parameters for Llama3-8B at rank 1).