  Queueing systems are widely applicable stochastic models with use cases in
communication networks, healthcare, service systems, etc. Although their
optimal control has been extensively studied, most existing approaches assume
perfect knowledge of the system parameters. Of course, this assumption rarely
holds in practice where there is parameter uncertainty, thus motivating a
recent line of work on bandit learning for queueing systems. This nascent
stream of research focuses on the asymptotic performance of the proposed
algorithms.
  In this paper, we argue that an asymptotic metric, which focuses on
late-stage performance, is insufficient to capture the intrinsic statistical
complexity of learning in queueing systems which typically occurs in the early
stage. Instead, we propose the Cost of Learning in Queueing (CLQ), a new metric
that quantifies the maximum increase in time-averaged queue length caused by
parameter uncertainty. We characterize the CLQ of a single queue multi-server
system, and then extend these results to multi-queue multi-server systems and
networks of queues. In establishing our results, we propose a unified analysis
framework for CLQ that bridges Lyapunov and bandit analysis, provides
guarantees for a wide range of algorithms, and could be of independent
interest.
