  As cloud applications shift from monoliths to loosely coupled microservices,
application developers must decide how many compute resources (e.g., number of
replicated containers) to assign to each microservice within an application.
This decision affects both (1) the dollar cost to the application developer and
(2) the end-to-end latency perceived by the application user. Today, individual
microservices are autoscaled independently by adding VMs whenever
per-microservice CPU or memory utilization crosses a configurable threshold.
However, an application user's end-to-end latency consists of time spent on
multiple microservices and each microservice might need a different number of
VMs to achieve an overall end-to-end latency.
  We present COLA, an autoscaler for microservice-based applications, which
collectively allocates VMs to microservices with a global goal of minimizing
dollar cost while keeping end-to-end application latency under a given target.
Using 5 open-source applications, we compared COLA to several utilization and
machine learning based autoscalers. We evaluate COLA across different compute
settings on Google Kubernetes Engine (GKE) in which users manage compute
resources, GKE standard, and a new mode of operation in which the cloud
provider manages compute infrastructure, GKE Autopilot. COLA meets a desired
median or tail latency target on 53 of 63 workloads where it provides a cost
reduction of 19.3%, on average, over the next cheapest autoscaler. COLA is the
most cost effective autoscaling policy for 48 of these 53 workloads. The cost
savings from managing a cluster with COLA result in COLA paying for its
training cost in a few days. On smaller applications, for which we can
exhaustively search microservice configurations, we find that COLA is optimal
for 90% of cases and near optimal otherwise.
