  Inference serving is of great importance in deploying machine learning models
in real-world applications, ensuring efficient processing and quick responses
to inference requests. However, managing resources in these systems poses
significant challenges, particularly in maintaining performance under varying
and unpredictable workloads. Two primary scaling strategies, horizontal and
vertical scaling, offer different advantages and limitations. Horizontal
scaling adds more instances to handle increased loads but can suffer from cold
start issues and increased management complexity. Vertical scaling boosts the
capacity of existing instances, allowing for quicker responses but is limited
by hardware and model parallelization capabilities.
  This paper introduces Themis, a system designed to leverage the benefits of
both horizontal and vertical scaling in inference serving systems. Themis
employs a two-stage autoscaling strategy: initially using in-place vertical
scaling to handle workload surges and then switching to horizontal scaling to
optimize resource efficiency once the workload stabilizes. The system profiles
the processing latency of deep learning models, calculates queuing delays, and
employs different dynamic programming algorithms to solve the joint horizontal
and vertical scaling problem optimally based on the workload situation.
Extensive evaluations with real-world workload traces demonstrate over
$10\times$ SLO violation reduction compared to the state-of-the-art horizontal
or vertical autoscaling approaches while maintaining resource efficiency when
the workload is stable.
