On the Statistical Mechanics of Attention
π arXiv:2610.17347 Β· π₯ PDF Β· 2026-10-12 Β· cond-mat.stat-mech
Authors: Jerry Yao-Chieh Hu [arXiv Β· scholar] , Yu-An Chen [arXiv Β· scholar] , Maojiang Su [arXiv Β· scholar] , Han Liu [arXiv Β· scholar]
π Abstract
We revisit the statistical mechanics of Transformer attention via modern Hopfield models. We present a free-energy formalism to characterize parameter regimes and critical thresholds for robust attention inference. The resulting free-energy landscape incorporates randomness in query states and attention weights. Attention then emerges as the equation of motion of the corresponding statistical-mechanical system (i.e., modern Hopfield model) through the standard free-energy minimization principle. This allows us to identify first-order phase transitions in attention between glassy and robust-inference phases. We further extend to attentions with stochastic queries and keys. Together, our results provide a rigorous statistical-mechanics framework for characterizing phase structure and inference reliability across a family of Transformer attention mechanisms.