论文：非线性注意力中的涌现逆深度缩放

1. 12:00
   arXiv · Machine Learning Theory

   注意力非线性如何催生逆深度缩放

   论文提出，注意力机制的非线性可让损失随模型深度呈逆深度衰减，在所有测试数据谱中均成立。非线性使注意力能选择性聚焦相关 token，强弱谱方向可并行学习；层间共享误差决定损失平台，层间差异聚合则随深度持续增益。该研究认为深度缩放可能源于注意力的非线性，使大语言模型实现局部聚焦。

Current heat 9·Comparable peak 10（Oct 9）·Comparable change over 24 hours –

The trend compares only the same participants observed continuously; its range may be smaller than the current heat count. Move or click on the chart to inspect hourly heat; use the left and right arrow keys to switch.