  A service-level objective (SLO) is a target performance metric of service
that cloud vendors aim to ensure. Delivering optimized SLOs can enhance user
satisfaction and improve the competitiveness of cloud vendors. As large
language models (LLMs) are gaining increasing popularity across various fields,
it is of great significance to optimize SLOs for LLM inference services. In
this paper, we observe that adjusting the parameters of LLM inference engines
can improve service performance, and the optimal parameter configurations of
different services are different. Therefore, we propose SCOOT, an automatic
performance tuning system to optimize SLOs for each LLM inference service by
tuning the parameters of the inference engine. We first propose a generalized
formulation of the tuning problem to handle various objectives and constraints
between parameters, and SCOOT exploits the Bayesian optimization (BO) technique
to resolve the problem via exploration and exploitation. Moreover, SCOOT adopts
a random forest to learn hidden constraints during the tuning process to
mitigate invalid exploration. To improve the tuning efficiency, SCOOT utilizes
the parallel suggestion to accelerate the tuning process. Extensive experiments
demonstrate that SCOOT can significantly outperform existing tuning techniques
in SLO optimization while greatly improving the tuning efficiency.
