  The development of mobile and on the edge applications that embed deep
convolutional neural models has the potential to revolutionise biomedicine.
However, most deep learning models require computational resources that are not
available in smartphones or edge devices; an issue that can be faced by means
of compact models. The problem with such models is that they are, at least
usually, less accurate than bigger models. In this work, we study how this
limitation can be addressed with the application of semi-supervised learning
techniques. We conduct several statistical analyses to compare performance of
deep compact architectures when trained using semi-supervised learning methods
for tackling image classification tasks in the biomedical context. In
particular, we explore three families of compact networks, and two families of
semi-supervised learning techniques for 10 biomedical tasks. By combining
semi-supervised learning methods with compact networks, it is possible to
obtain a similar performance to standard size networks. In general, the best
results are obtained when combining data distillation with MixNet, and plain
distillation with ResNet-18. Also, in general, NAS networks obtain better
results than manually designed networks and quantized networks. The work
presented in this paper shows the benefits of apply semi-supervised methods to
compact networks; this allow us to create compact models that are not only as
accurate as standard size models, but also faster and lighter. Finally, we have
developed a library that simplifies the construction of compact models using
semi-supervised learning methods.
