  In semi-supervised representation learning frameworks, when the number of
labelled data is very scarce, the quality and representativeness of these
samples become increasingly important. Existing literature on semi-supervised
learning randomly sample a limited number of data points for labelling. All
these labelled samples are then used along with the unlabelled data throughout
the training process. In this work, we ask two important questions in this
context: (1) does it matter which samples are selected for labelling? (2) does
it matter how the labelled samples are used throughout the training process
along with the unlabelled data? To answer the first question, we explore a
number of unsupervised methods for selecting specific subsets of data to label
(without prior knowledge of their labels), with the goal of maximizing
representativeness w.r.t. the unlabelled set. Then, for our second line of
inquiry, we define a variety of different label injection strategies in the
training process. Extensive experiments on four popular datasets, CIFAR-10,
CIFAR-100, SVHN, and STL-10, show that unsupervised selection of samples that
are more representative of the entire data improves performance by up to ~2%
over the existing semi-supervised frameworks such as MixMatch, ReMixMatch,
FixMatch and others with random sample labelling. We show that this boost could
even increase to 7.5% for very few-labelled scenarios. However, our study shows
that gradually injecting the labels throughout the training procedure does not
impact the performance considerably versus when all the existing labels are
used throughout the entire training.
