  With the development of deep convolutional neural networks, medical image
segmentation has achieved a series of breakthroughs in recent years. However,
the high-performance convolutional neural networks always mean numerous
parameters and high computation costs, which will hinder the applications in
clinical scenarios. Meanwhile, the scarceness of large-scale annotated medical
image datasets further impedes the application of high-performance networks. To
tackle these problems, we propose Graph Flow, a comprehensive knowledge
distillation framework, for both network-efficiency and annotation-efficiency
medical image segmentation. Specifically, our core Graph Flow Distillation
transfer the essence of cross-layer variations from a well-trained cumbersome
teacher network to a non-trained compact student network. In addition, an
unsupervised Paraphraser Module is integrated to purify the knowledge of the
teacher network, which is also beneficial for the stabilization of training
procedure. Furthermore, we build a unified distillation framework by
integrating the adversarial distillation and the vanilla logits distillation,
which can further refine the final predictions of the compact network. With
different teacher networks (conventional convolutional architecture or
prevalent transformer architecture) and student networks, we conduct extensive
experiments on four medical image datasets with different modalities (Gastric
Cancer, Synapse, BUSI, and CVC-ClinicDB).We demonstrate the prominent ability
of our method which achieves competitive performance on these datasets.
Moreover, we demonstrate the effectiveness of our Graph Flow through a novel
semi-supervised paradigm for dual efficient medical image segmentation. Our
code will be available at Graph Flow.
