Differential Attention: A New Mechanism That Reduces Hallucinations by 60%

Researchers at UC Berkeley and Google DeepMind have published a paper introducing Differential Attention, a novel attention mechanism that computes the difference between two softmax attention maps. This approach effectively cancels out noise and reduces hallucinations by 60% on TruthfulQA benchmarks while maintaining comparable computational costs to standard multi-head attention.