Why Gradient Descent Zigzags and How Momentum Fixes It

A stretched quadratic loss surface shows why vanilla gradient descent can struggle on uneven curvature.

It overshoots in the steep direction and moves slowly along the flat direction, creating zigzagging updates.

Momentum fixes this by keeping a running average of gradients, which dampens oscillations and builds useful velocity.

In simulation, the momentum optimizer converges faster and more smoothly than plain gradient descent.