  Convolutional neural networks trained without supervision come close to
matching performance with supervised pre-training, but sometimes at the cost of
an even higher number of parameters. Extracting subnetworks from these large
unsupervised convnets with preserved performance is of particular interest to
make them less computationally intensive. Typical pruning methods operate
during training on a task while trying to maintain the performance of the
pruned network on the same task. However, in self-supervised feature learning,
the training objective is agnostic on the representation transferability to
downstream tasks. Thus, preserving performance for this objective does not
ensure that the pruned subnetwork remains effective for solving downstream
tasks. In this work, we investigate the use of standard pruning methods,
developed primarily for supervised learning, for networks trained without
labels (i.e. on self-supervised tasks). We show that pruned masks obtained with
or without labels reach comparable performance when re-trained on labels,
suggesting that pruning operates similarly for self-supervised and supervised
learning. Interestingly, we also find that pruning preserves the transfer
performance of self-supervised subnetwork representations.
