Deep Compression on Segment Anything Model for Efficient Industrial Manufacturing

Segment Anything Model (SAM) is a popular vision foundation model that can segment data from any domain. Benefiting from its outstanding generalization ability, SAM has been widely adopted in many industrial scenarios. However, as SAM is built upon a heavy Vision Transformer (ViT), it suffers from memory-hungry and low latency, which restricts the deployment on edge devices. In this paper, we systematically explore how to compress SAM effectively, making it feasible to adapt edge devices with limited calculation abilities. Specifically, our method consists of three aspects: weight initialization, knowledge distillation, and model quantization. It is notable that all three aspects are not simply inherited from previous methods, but tactfully designed based on the teacher-student learning paradigm, considering the task-attributes of SAM pre-training. Firstly, we design a weight initialization method for fully using the pre-training knowledge implicitly contained in the teacher’s parameter space. Secondly, based on the weight initialization, we design a novel distillation method tailored to SAM pre-training, focusing on learning the semantic differences among areas. Lastly, we perform quantization on our distilled models. Unlike the previous method, we used both the teacher and the student to calibrate our model in the quantization process. We conduct systematic experiments on various teacher-student network pairs to validate the broad effectiveness of our method. By applying our method, our target models achieve over 64.5× speed increase compared to the original SAM. Core code is available at: https://github.com/ZG-ZZ/DC-SAM.