  Compact convolutional neural networks (CNNs) have witnessed exceptional
improvements in performance in recent years. However, they still fail to
provide the same predictive power as CNNs with a large number of parameters.
The diverse and even abundant features captured by the layers is an important
characteristic of these successful CNNs. However, differences in this
characteristic between large CNNs and their compact counterparts have rarely
been investigated. In compact CNNs, due to the limited number of parameters,
abundant features are unlikely to be obtained, and feature diversity becomes an
essential characteristic. Diverse features present in the activation maps
derived from a data point during model inference may indicate the presence of a
set of unique descriptors necessary to distinguish between objects of different
classes. In contrast, data points with low feature diversity may not provide a
sufficient amount of unique descriptors to make a valid prediction; we refer to
them as random predictions. Random predictions can negatively impact the
optimization process and harm the final performance. This paper proposes
addressing the problem raised by random predictions by reshaping the standard
cross-entropy to make it biased toward data points with a limited number of
unique descriptive features. Our novel Bias Loss focuses the training on a set
of valuable data points and prevents the vast number of samples with poor
learning features from misleading the optimization process. Furthermore, to
show the importance of diversity, we present a family of SkipNet models whose
architectures are brought to boost the number of unique descriptors in the last
layers. Our Skipnet-M can achieve 1% higher classification accuracy than
MobileNetV3 Large.
