  Knowledge distillation has become an important approach to obtain a compact
yet effective model. To achieve this goal, a small student model is trained to
exploit the knowledge of a large well-trained teacher model. However, due to
the capacity gap between the teacher and the student, the student's performance
is hard to reach the level of the teacher. Regarding this issue, existing
methods propose to reduce the difficulty of the teacher's knowledge via a proxy
way. We argue that these proxy-based methods overlook the knowledge loss of the
teacher, which may cause the student to encounter capacity bottlenecks. In this
paper, we alleviate the capacity gap problem from a new perspective with the
purpose of averting knowledge loss. Instead of sacrificing part of the
teacher's knowledge, we propose to build a more powerful student via
adversarial collaborative learning. To this end, we further propose an
Adversarial Collaborative Knowledge Distillation (ACKD) method that effectively
improves the performance of knowledge distillation. Specifically, we construct
the student model with multiple auxiliary learners. Meanwhile, we devise an
adversarial collaborative module (ACM) that introduces attention mechanism and
adversarial learning to enhance the capacity of the student. Extensive
experiments on four classification tasks show the superiority of the proposed
ACKD.
