Shallow-to-Deep and Slack Matching for Self-Distillation

Self-knowledge distillation (self-KD) is an attractive technique that empowers the student to distill knowledge within itself, in which one predominant self-KD scheme is to teach the shallow layers with the deepest layer. However, such a paradigm causes the deepest classifier, the one to be deployed, to miss the opportunity to utilize the beneficial local clues and finer details encoded in the shallow layers. Besides, the existing KD manners enforce an overambitious exact value matching, suppressing the knowledge transfer. To deal with these limitations, we put forth Shallow-to-Deep and Slack Matching (SDSM), a novel approach for self-KD. In particular, SDSM reverses the teaching flow from deep-to-shallow into shallow-to-deep, thereby allowing the final classifier to explicitly assemble hierarchical knowledge contained in different model depths. Moreover, towards slack self-KD, we propose comprehensive attention distillation (CAD) and shift variance distillation (SVD) for feature-level and logitlevel self-KD, respectively. Leveraging the maximum operation, CAD integrates the class attention map from the shallow layers to excavate and transfer the beneficial information into the deepest layer. SVD, building upon the translation-invariance of softmax, minimizes the variance of logit shift, which effectively eliminates the concern regarding the logit’s value magnitude during distillation. Over several model architectures and benchmarks, extensive experiments demonstrate that SDSM consistently surpasses the prior competitors, manifesting its effectiveness and superiority.