  Knowledge distillation (KD) has been actively studied for image
classification tasks in deep learning, aiming to improve the performance of a
student based on the knowledge from a teacher. However, applying KD in image
regression with a scalar response variable has been rarely studied, and there
exists no KD method applicable to both classification and regression tasks yet.
Moreover, existing KD methods often require a practitioner to carefully select
or adjust the teacher and student architectures, making these methods less
flexible in practice. To address the above problems in a unified way, we
propose a comprehensive KD framework based on cGANs, termed cGAN-KD.
Fundamentally different from existing KD methods, cGAN-KD distills and
transfers knowledge from a teacher model to a student model via cGAN-generated
samples. This novel mechanism makes cGAN-KD suitable for both classification
and regression tasks, compatible with other KD methods, and insensitive to the
teacher and student architectures. An error bound for a student model trained
in the cGAN-KD framework is derived in this work, providing a theory for why
cGAN-KD is effective as well as guiding the practical implementation of
cGAN-KD. Extensive experiments on CIFAR-100 and ImageNet-100 show that we can
combine state of the art KD methods with the cGAN-KD framework to yield a new
state of the art. Moreover, experiments on Steering Angle and UTKFace
demonstrate the effectiveness of cGAN-KD in image regression tasks, where
existing KD methods are inapplicable.
