MoRA：基于路由偏差学习与专家近似的MoE剪枝框架

1. 12:00
   arXiv · Machine Learning Theory

   MoRA: 通过路由器偏置学习与专家近似实现 MoE 剪枝

   MoRA 提出一种结构化 MoE 专家剪枝框架，为每个专家引入可学习路由器偏置，通过最小化语言建模损失与路由多样性正则化来识别关键专家并鼓励多样化路由。剪枝后利用剩余专家通过仿射变换近似被剪除专家的输出，进一步提升性能。

Current heat 9·Comparable peak 10（Oct 2）·Comparable change over 24 hours –

The trend compares only the same participants observed continuously; its range may be smaller than the current heat count. Move or click on the chart to inspect hourly heat; use the left and right arrow keys to switch.