  Pruning neural networks, i.e., removing some of their parameters whilst
retaining their accuracy, is one of the main ways to reduce the latency of a
machine learning pipeline, especially in resource- and/or bandwidth-constrained
scenarios. In this context, the pruning technique, i.e., how to choose the
parameters to remove, is critical to the system performance. In this paper, we
propose a novel pruning approach, called FlexRel and predicated upon combining
training-time and inference-time information, namely, parameter magnitude and
relevance, in order to improve the resulting accuracy whilst saving both
computational resources and bandwidth. Our performance evaluation shows that
FlexRel is able to achieve higher pruning factors, saving over 35% bandwidth
for typical accuracy targets.
