Surpassing Oneself: Self-Distillation From Past Failures

Self-distillation improves model performance without external teachers, yet most existing methods rely on temporal averaging or consistency, which often reinforces errors and limits convergence. We propose a self-distillation framework that explicitly learns from past failures. In each round, the previous student provides experiences that are decomposed into successes and failures. Successful experiences are directly transferred as reliable knowledge, while failed experiences are transformed into useful guidance through a principled correction mechanism that enforces a clear margin between the ground-truth class and the dominant misclassified class. To enhance efficiency and stability, we introduce a dynamic curriculum strategy inspired by the "easy-to-hard" learning paradigm: the student first consolidates representations by focusing on successful experiences, and then gradually increases reliance on corrected failures as its capacity improves. Experiments on CIFAR-100 [1], Tiny-ImageNet [2] and CUB-200 [3] show consistent improvements over representative self-distillation methods, with both higher accuracy and greater robustness in small-sample and confusion-prone scenarios. Overall, our approach enables students to learn from both successes and failures, driving continuous self-improvement.