  With the rapid development of computer vision, Vision Transformers (ViTs)
offer the tantalizing prospect of unified information processing across visual
and textual domains. But due to the lack of inherent inductive biases in ViTs,
they require enormous amount of data for training. To make their applications
practical, we introduce an innovative ensemble-based distillation approach
distilling inductive bias from complementary lightweight teacher models. Prior
systems relied solely on convolution-based teaching. However, this method
incorporates an ensemble of light teachers with different architectural
tendencies, such as convolution and involution, to instruct the student
transformer jointly. Because of these unique inductive biases, instructors can
accumulate a wide range of knowledge, even from readily identifiable stored
datasets, which leads to enhanced student performance. Our proposed framework
also involves precomputing and storing logits in advance, essentially the
unnormalized predictions of the model. This optimization can accelerate the
distillation process by eliminating the need for repeated forward passes during
knowledge distillation, significantly reducing the computational burden and
enhancing efficiency.
