  When deploying machine learning (ML) applications, the automated allocation
of computing resources-commonly referred to as autoscaling-is crucial for
maintaining a consistent inference time under fluctuating workloads. The
objective is to maximize the Quality of Service metrics, emphasizing
performance and availability, while minimizing resource costs. In this paper,
we compare scalable deployment techniques across three levels of scaling: at
the application level (TorchServe, RayServe) and the container level (K3s) in a
local environment (production server), as well as at the container and machine
levels in a cloud environment (Amazon Web Services Elastic Container Service
and Elastic Kubernetes Service). The comparison is conducted through the study
of mean and standard deviation of inference time in a multi-client scenario,
along with upscaling response times. Based on this analysis, we propose a
deployment strategy for both local and cloud-based environments.
