Compression Techniques for Large-Scale Transformer Models: A Comprehensive Survey

In recent years, deep learning models based on the Transformer architecture have achieved remarkable success in natural language processing and computer vision. However, these models typically contain hundreds of millions or even billions of parameters, posing enormous computational and storage challenges for practical deployment. Model compression technology has become a key solution to this problem by reducing model redundancy and lowering computational resource requirements while maintaining performance. This paper systematically reviews the four core technologies of Transformer model compression: model quantization, knowledge distillation, model pruning, and low-rank approximation, and analyzes in detail the principles, characteristics, and applications of various methods in vision Transformers and large language models, as well as discusses the future development directions of this field.