arXiv · Statistics Machine Learning· Jiangfeng Chen, Xinyu Wang, Tianshuo Yan, Hanwei Wu, Xiao-Wen Chang, Yang Zhang, Lei Ding·· 3 小时前AI 评分28
Sequential Functional Structured Tucker Compression for Large Language Model Attentions
Sequential Functional Structured Tucker Compression for Large Language Model Attentions
AI 导读
FTC是一种序贯结构化压缩框架，在固定存储预算下适配当前压缩模型并联合利用Q/K/V头结构，输出投影单独处理以应对表示变化。实验覆盖6B至32B七个解码器仅LLM，在五款现代GQA模型的WikiText-2困惑度上各keep ratio均最优，激进压缩下增益最大，且改进迁移到下游任务并在32B规模仍显著。无需微调或基于梯度的恢复。
来源：arXiv · Statistics Machine Learning · arxiv.org