Sparse Mixture of Attention Heads Enables 10x Context Length Scaling

A new paper from DeepMind introduces Sparse Mixture of Attention (SMoA), which dynamically routes tokens to specialized attention heads. This allows models to process context windows up to 10x longer than standard transformers without proportional increases in compute. The technique shows particular promise for document understanding and multi-turn conversation tasks.