  MLaaS (ML-as-a-Service) offerings by cloud computing platforms are becoming
increasingly popular. Hosting pre-trained machine learning models in the cloud
enables elastic scalability as the demand grows. But providing low latency and
reducing the latency variance is a key requirement. Variance is harder to
control in a cloud deployment due to uncertainties in resource allocations
across many virtual instances. We propose the collage inference technique which
uses a novel convolutional neural network model, collage-cnn, to provide
low-cost redundancy. A collage-cnn model takes a collage image formed by
combining multiple images and performs multi-image classification in one shot,
albeit at slightly lower accuracy. We augment a collection of traditional
single image classifier models with a single collage-cnn classifier which acts
as their low-cost redundant backup. Collage-cnn provides backup classification
results if any single image classification requests experience slowdown.
Deploying the collage-cnn models in the cloud, we demonstrate that the 99th
percentile tail latency of inference can be reduced by 1.2x to 2x compared to
replication based approaches while providing high accuracy. Variation in
inference latency can be reduced by 1.8x to 15x.
