New Optimisers.
Treat a weight matrix as a matrix, not a bag of numbers — and stop a few directions from taking over every step.
Adam rescales each parameter by its own gradient history, ignoring that the parameters of a layer form a matrix. The gradient of a weight matrix is typically dominated by a few directions — its singular values are very uneven — so an Adam step spends most of its size on those few. Second-order and matrix-aware methods try to do better. Shampoo keeps running statistics of the gradient’s row and column covariances and preconditions with their inverse roots; a distributed implementation of it won the first AlgoPerf benchmark of training speed in 2024.
Muon (2024) is simpler: take the momentum of a weight matrix’s gradient, and replace it with the nearest orthogonal matrix — same singular vectors, all singular values set to one — computed approximately by a few steps of a Newton–Schulz iteration. Every direction in the update then moves at the same rate. Moonshot AI reported that Muon reached the same loss as AdamW with about half the compute in scaling-law experiments, and trained Kimi K2, a trillion-parameter model, with a variant. The workshop shows what the orthogonalisation does to an update’s singular values.
Below: left, the singular values of an 8 × 8 update matrix with a 100-to-1 spread, before and after the chosen number of Newton–Schulz steps. Right, published results for matrix-aware optimisers.
- Explain why matrix structure matters for an optimiser.
- Describe Muon’s orthogonalised update.
- Describe Shampoo’s preconditioning.
- Read the published evidence for each.
Equal rates for every direction
The update matrix starts with singular values spread 100 to 1: its largest direction would take a hundred times the step of its smallest. After 5 Newton–Schulz steps they lie between 0.70 and 1.13; after one step, only the largest have been pulled up. The iteration is not exact orthogonalisation — Muon’s coefficients are chosen to push singular values into a band near one quickly rather than precisely — but it costs only a few matrix multiplications, small next to the forward and backward passes, and runs in bfloat16.
The evidence for such optimisers has grown quickly. Distributed Shampoo scored 0.78 in AlgoPerf’s external-tuning track against a baseline of 0.57, showing that non-diagonal preconditioning can beat Adam on wall-clock time. Moonshot AI reported Muon at about 2 times AdamW’s compute efficiency and trained Moonlight and then Kimi K2 with it, adding a clip on attention logits to keep it stable (plate XI.5). Adam remains the default for most runs; whether matrix-aware methods replace it at the largest scales is an open and active question.
Terms
- Preconditioning
- Transforming the gradient by an estimate of the loss surface’s curvature before stepping.
- Shampoo
- An optimiser that preconditions matrix gradients with inverse roots of their row and column statistics.
- Muon
- An optimiser that replaces each matrix update’s momentum with an approximately orthogonal matrix.
- Newton–Schulz iteration
- A matrix polynomial iteration that drives singular values towards one.
Sources
- Jordan et al., “Muon: An optimizer for hidden layers in neural networks”, 2024 (blog post and code).
- Liu et al., “Muon is Scalable for LLM Training”, arXiv:2502.16982, 2025.
- Gupta, Koren and Singer, “Shampoo: Preconditioned Stochastic Tensor Optimization”, ICML 2018.
- Kasimbeg et al., “Accelerating Neural Network Training: An Analysis of the AlgoPerf Competition”, ICLR 2025.
- Kimi Team, “Kimi K2: Open Agentic Intelligence”, technical report, July 2025.