DIVE into MoE: Diversity-Enhanced Reconstruction of Large Language Models from Dense into Mixture-of-Experts
Yuchen Feng, Bowen Shen, Naibin Gu, Jiaxuan Zhao, Peng Fu, Zheng Lin, Weiping Wang
Chinese Academy of Sciences Institute of Information Engineering University of Chinese Academy of Sciences
阅读操作
确认中在文库中上传 PDF 后可生成中文音频讲解。
摘要与影响
Large language models (LLMs) with the Mixture-of-Experts (MoE) architecture achieve high cost-efficiency by selectively activating a subset of the parameters.Despite the inference efficiency of MoE LLMs, the training of extensive experts from scratch incurs substantial overhead, whereas reconstructing a dense LLM into an MoE LLM significantly reduces the training budget.However, existing reconstruction methods often overlook the diversity among experts, leading to potential redundancy.In this paper, we come up with the observation that a specific LLM exhibits notable diversity after being pruned on different calibration datasets, based on which we present a Diversity-Enhanced reconstruction method named DIVE.The recipe of DIVE includes domain affinity mining, pruning-based expert reconstruction, and efficient retraining.Specifically, the reconstruction includes pruning and reassembly of the feed-forward network (FFN) module.After reconstruction, we efficiently retrain the model on routers, experts and normalization modules.We implement DIVE on Llama-style LLMs with open-source training corpora.Experiments show that DIVE achieves training efficiency with minimal accuracy tradeoffs, outperforming existing pruning and MoE reconstruction methods with the same number of activated parameters.Code is available at
逐年被引趋势
关键指标
同类平均 = 1
同领域 · 同年份 · 同类型
Google Scholar 与 OpenAlex 的被引统计范围不同,数值存在差异属正常。
AI 辅助阅读
依据:摘要
可就本文提问;依据不足时会说明。
学术脉络
学科主题
计算机 / AIAdvanced Neural Network Applications
Topic Modeling · Natural Language Processing Techniques