  Recent advancements in language models (LMs) have gained substantial
attentions on their capability to generate human-like responses. Though
exhibiting a promising future for various applications such as conversation AI,
these LMs face deployment challenges on various devices due to their extreme
computational cost and unpredictable inference latency. Such varied inference
latency, identified as a consequence of uncertainty intrinsic to the nature of
language, can lead to computational inefficiency and degrade the overall
performance of LMs, especially under high-traffic workloads. Unfortunately, the
bandwidth of these uncertainty sources is extensive, complicating the
prediction of latency and the effects emanating from such uncertainties. To
understand and mitigate the impact of uncertainty on real-time
response-demanding systems, we take the first step to comprehend, quantify and
optimize these uncertainty-induced latency performance variations in LMs.
Specifically, we present RT-LM, an uncertainty-aware resource management
ecosystem for real-time inference of LMs. RT-LM innovatively quantifies how
specific input uncertainties, adversely affect latency, often leading to an
increased output length. Exploiting these insights, we devise a lightweight yet
effective method to dynamically correlate input text uncertainties with output
length at runtime. Utilizing this quantification as a latency heuristic, we
integrate the uncertainty information into a system-level scheduler which
explores several uncertainty-induced optimization opportunities, including
uncertainty-aware prioritization, dynamic consolidation, and strategic CPU
offloading. Quantitative experiments across five state-of-the-art LMs on two
hardware platforms demonstrates that RT-LM can significantly reduce the average
response time and improve throughput while incurring a rather small runtime
overhead.
