Tag: optimizers

All the articles with the tag "optimizers".

• Why Adam won

  Published:Jul 25, 2026 |  at 08:40 PM

  One level deeper on Adam: per-parameter steps, bias correction, the sign-descent view, why it wins on language, and AdamW. With a short NumPy version.

• Optimizers: SGD to Adam to what's next

  Published:Jul 18, 2026 |  at 09:20 PM

  How a network takes a step downhill, from gradient descent and momentum to Adam, AdamW, Muon and Shampoo, and which ones big labs really use.