  Deep Neural Networks (DNNs) have advanced the state-of-the-art in a variety
of machine learning tasks and are deployed in increasing numbers of products
and services. However, the computational requirements of training and
evaluating large-scale DNNs are growing at a much faster pace than the
capabilities of the underlying hardware platforms that they are executed upon.
In this work, we propose Dynamic Variable Effort Deep Neural Networks
(DyVEDeep) to reduce the computational requirements of DNNs during inference.
Previous efforts propose specialized hardware implementations for DNNs,
statically prune the network, or compress the weights. Complementary to these
approaches, DyVEDeep is a dynamic approach that exploits the heterogeneity in
the inputs to DNNs to improve their compute efficiency with comparable
classification accuracy. DyVEDeep equips DNNs with dynamic effort mechanisms
that, in the course of processing an input, identify how critical a group of
computations are to classify the input. DyVEDeep dynamically focuses its
compute effort only on the critical computa- tions, while skipping or
approximating the rest. We propose 3 effort knobs that operate at different
levels of granularity viz. neuron, feature and layer levels. We build DyVEDeep
versions for 5 popular image recognition benchmarks - one for CIFAR-10 and four
for ImageNet (AlexNet, OverFeat and VGG-16, weight-compressed AlexNet). Across
all benchmarks, DyVEDeep achieves 2.1x-2.6x reduction in the number of scalar
operations, which translates to 1.8x-2.3x performance improvement over a
Caffe-based implementation, with < 0.5% loss in accuracy.
