  Deep learning models have been used to support analytics beyond simple
aggregation, where deeper and wider models have been shown to yield great
results. These models consume a huge amount of memory and computational
operations. However, most of the large-scale industrial applications are often
computational budget constrained. In practice, the peak workload of inference
service could be 10x higher than the average cases, with the presence of
unpredictable extreme cases. Lots of computational resources could be wasted
during off-peak hours and the system may crash when the workload exceeds system
capacity. How to support deep learning services with a dynamic workload
cost-efficiently remains a challenging problem. In this paper, we address the
challenge with a general and novel training scheme called model slicing, which
enables deep learning models to provide predictions within the prescribed
computational resource budget dynamically. Model slicing could be viewed as an
elastic computation solution without requiring more computational resources.
Succinctly, each layer in the model is divided into groups of a contiguous
block of basic components (i.e. neurons in dense layers and channels in
convolutional layers), and then partially ordered relation is introduced to
these groups by enforcing that groups participated in each forward pass always
starts from the first group to the dynamically-determined rightmost group.
Trained by dynamically indexing the rightmost group with a single parameter
slice rate, the network is engendered to build up group-wise and residual
representation. Then during inference, a sub-model with fewer groups can be
readily deployed for efficiency whose computation is roughly quadratic to the
width controlled by the slice rate. Extensive experiments show that models
trained with model slicing can effectively support on-demand workload with
elastic inference cost.
