  Generative adversarial networks (GANs) have shown significant potential in
modeling high dimensional distributions of image data, especially on
image-to-image translation tasks. However, due to the complexity of these
tasks, state-of-the-art models often contain a tremendous amount of parameters,
which results in large model size and long inference time. In this work, we
propose a novel method to address this problem by applying knowledge
distillation together with distillation of a semantic relation preserving
matrix. This matrix, derived from the teacher's feature encoding, helps the
student model learn better semantic relations. In contrast to existing
compression methods designed for classification tasks, our proposed method
adapts well to the image-to-image translation task on GANs. Experiments
conducted on 5 different datasets and 3 different pairs of teacher and student
models provide strong evidence that our methods achieve impressive results both
qualitatively and quantitatively.
