  Serving large language models (LLMs) in production can incur substantial
costs, which has prompted recent advances in inference system optimizations.
Today, these systems are evaluated against conventional latency and throughput
metrics (eg. TTFT, TBT, Normalised Latency and TPOT). However, these metrics
fail to fully capture the nuances of LLM inference, leading to an incomplete
assessment of user-facing performance crucial for real-time applications such
as chat and translation. In this paper, we first identify the pitfalls of
current performance metrics in evaluating LLM inference systems. We then
propose Metron, a comprehensive performance evaluation framework that includes
fluidity-index -- a novel metric designed to reflect the intricacies of the LLM
inference process and its impact on real-time user experience. Finally, we
evaluate various existing open-source platforms and model-as-a-service
offerings using Metron, discussing their strengths and weaknesses. Metron is
available at https://github.com/project-metron/metron.
