  Pruning - that is, setting a significant subset of the parameters of a neural
network to zero - is one of the most popular methods of model compression. Yet,
several recent works have raised the issue that pruning may induce or
exacerbate bias in the output of the compressed model. Despite existing
evidence for this phenomenon, the relationship between neural network pruning
and induced bias is not well-understood. In this work, we systematically
investigate and characterize this phenomenon in Convolutional Neural Networks
for computer vision. First, we show that it is in fact possible to obtain
highly-sparse models, e.g. with less than 10% remaining weights, which do not
decrease in accuracy nor substantially increase in bias when compared to dense
models. At the same time, we also find that, at higher sparsities, pruned
models exhibit higher uncertainty in their outputs, as well as increased
correlations, which we directly link to increased bias. We propose easy-to-use
criteria which, based only on the uncompressed model, establish whether bias
will increase with pruning, and identify the samples most susceptible to biased
predictions post-compression.
