  Intelligent medical diagnosis has shown remarkable progress based on the
large-scale datasets with precise annotations. However, fewer labeled images
are available due to significantly expensive cost for annotating data by
experts. To fully exploit the easily available unlabeled data, we propose a
novel Spatio-Temporal Structure Consistent (STSC) learning framework.
Specifically, a gram matrix is derived to combine the spatial structure
consistency and temporal structure consistency together. This gram matrix
captures the structural similarity among the representations of different
training samples. At the spatial level, our framework explicitly enforces the
consistency of structural similarity among different samples under
perturbations. At the temporal level, we consider the consistency of the
structural similarity in different training iterations by digging out the
stable sub-structures in a relation graph. Experiments on two medical image
datasets (i.e., ISIC 2018 challenge and ChestX-ray14) show that our method
outperforms state-of-the-art SSL methods. Furthermore, extensive qualitative
analysis on the Gram matrices and heatmaps by Grad-CAM are presented to
validate the effectiveness of our method.
