  The recent success of Deep Neural Networks (DNNs) has drastically improved
the state of the art for many application domains. While achieving high
accuracy performance, deploying state-of-the-art DNNs is a challenge since they
typically require billions of expensive arithmetic computations. In addition,
DNNs are typically deployed in ensemble to boost accuracy performance, which
further exacerbates the system requirements. This computational overhead is an
issue for many platforms, e.g. data centers and embedded systems, with tight
latency and energy budgets. In this article, we introduce flexible DNNs
ensemble processing technique, which achieves large reduction in average
inference latency while incurring small to negligible accuracy drop. Our
technique is flexible in that it allows for dynamic adaptation between quality
of results (QoR) and execution runtime. We demonstrate the effectiveness of the
technique on AlexNet and ResNet-50 using the ImageNet dataset. This technique
can also easily handle other types of networks.
