  This paper proposes a content relationship distillation (CRD) to tackle the
over-parameterized generative adversarial networks (GANs) for the
serviceability in cutting-edge devices. In contrast to traditional
instance-level distillation, we design a novel GAN compression oriented
knowledge by slicing the contents of teacher outputs into multiple fine-grained
granularities, such as row/column strips (global information) and image patches
(local information), modeling the relationships among them, such as pairwise
distance and triplet-wise angle, and encouraging the student to capture these
relationships within its output contents. Built upon our proposed content-level
distillation, we also deploy an online teacher discriminator, which keeps
updating when co-trained with the teacher generator and keeps freezing when
co-trained with the student generator for better adversarial training. We
perform extensive experiments on three benchmark datasets, the results of which
show that our CRD reaches the most complexity reduction on GANs while obtaining
the best performance in comparison with existing methods. For example, we
reduce MACs of CycleGAN by around 40x and parameters by over 80x, meanwhile,
46.61 FIDs are obtained compared with these of 51.92 for the current
state-of-the-art. Code of this project is available at
https://github.com/TheKernelZ/CRD.
