  Reducing the latency variance in machine learning inference is a key
requirement in many applications. Variance is harder to control in a cloud
deployment in the presence of stragglers. In spite of this challenge, inference
is increasingly being done in the cloud, due to the advent of affordable
machine learning as a service (MLaaS) platforms. Existing approaches to reduce
variance rely on replication which is expensive and partially negates the
affordability of MLaaS. In this work, we argue that MLaaS platforms also
provide unique opportunities to cut the cost of redundancy. In MLaaS platforms,
multiple inference requests are concurrently received by a load balancer which
can then create a more cost-efficient redundancy coding across a larger
collection of images. We propose a novel convolutional neural network model,
Collage-CNN, to provide a low-cost redundancy framework. A Collage-CNN model
takes a collage formed by combining multiple images and performs multi-image
classification in one shot, albeit at slightly lower accuracy. We then augment
a collection of traditional single image classifiers with a single Collage-CNN
classifier which acts as a low-cost redundant backup. Collage-CNN then provides
backup classification results if a single image classification straggles.
Deploying the Collage-CNN models in the cloud, we demonstrate that the 99th
percentile tail latency of inference can be reduced by 1.47X compared to
replication based approaches while providing high accuracy. Also, variation in
inference latency can be reduced by 9X with a slight increase in average
inference latency.
