  In this paper, we investigate how to push the performance limits of serving
Deep Neural Network (DNN) models on CPU-based servers. Specifically, we observe
that while intra-operator parallelism across multiple threads is an effective
way to reduce inference latency, it provides diminishing returns. Our primary
insight is that instead of running a single instance of a model with all
available threads on a server, running multiple instances each with smaller
batch sizes and fewer threads for intra-op parallelism can provide lower
inference latency. However, the right configuration is hard to determine
manually since it is workload- (DNN model and batch size used by the serving
system) and deployment-dependent (number of CPU cores on server). We present
Packrat, a new serving system for online inference that given a model and batch
size ($B$) algorithmically picks the optimal number of instances ($i$), the
number of threads each should be allocated ($t$), and the batch sizes each
should operate on ($b$) that minimizes latency. Packrat is built as an
extension to TorchServe and supports online reconfigurations to avoid serving
downtime. Averaged across a range of batch sizes, Packrat improves inference
latency by 1.43$\times$ to 1.83$\times$ on a range of commonly used DNNs.
