  Despite the promising results of convolutional neural networks (CNNs), their
application on devices with limited resources is still a big challenge; this is
mainly due to the huge memory and computation requirements of the CNN. To
counter the limitation imposed by the network size, we use pruning to reduce
the network size and -- implicitly -- the number of floating point operations
(FLOPs). Contrary to the filter norm method -- used in ``conventional`` network
pruning -- based on the assumption that a smaller norm implies ``less
importance'' to its associated component, we develop a novel filter importance
norm that is based on the change in the empirical loss caused by the presence
or removal of a component from the network architecture.
  Since there are too many individual possibilities for filter configuration,
we repeatedly sample from these architectural components and measure the system
performance in the respective state of components being active or disabled. The
result is a collection of filter ensembles -- filter masks -- and associated
performance values. We rank the filters based on a linear and additive model
and remove the least important ones such that the drop in network accuracy is
minimal. We evaluate our method on a fully connected network, as well as on the
ResNet architecture trained on the CIFAR-10 dataset. Using our pruning method,
we managed to remove $60\%$ of the parameters and $64\%$ of the FLOPs from the
ResNet with an accuracy drop of less than $0.6\%$.
