  Modern large language models (LLMs) represent a paradigm shift in what can
plausibly be expected of machine learning models. The fact that LLMs can
effectively generate sensible answers to a diverse range of queries suggests
that they would be useful in customer support applications. While powerful,
LLMs have been observed to be prone to hallucination which unfortunately makes
their near term use in customer support applications challenging. To address
this issue we present a system that allows us to use an LLM to augment our
customer support advocates by re-framing the language modeling task as a
discriminative classification task. In this framing, we seek to present the
top-K best template responses for a customer support advocate to use when
responding to a customer. We present the result of both offline and online
experiments where we observed offline gains and statistically significant
online lifts for our experimental system. Along the way, we present observed
scaling curves for validation loss and top-K accuracy, resulted from model
parameter ablation studies. We close by discussing the space of trade-offs with
respect to model size, latency, and accuracy as well as and suggesting future
applications to explore.
