  In this paper, we present Smart Compose, a novel system for generating
interactive, real-time suggestions in Gmail that assists users in writing mails
by reducing repetitive typing. In the design and deployment of such a
large-scale and complicated system, we faced several challenges including model
selection, performance evaluation, serving and other practical issues. At the
core of Smart Compose is a large-scale neural language model. We leveraged
state-of-the-art machine learning techniques for language model training which
enabled high-quality suggestion prediction, and constructed novel serving
infrastructure for high-throughput and real-time inference. Experimental
results show the effectiveness of our proposed system design and deployment
approach. This system is currently being served in Gmail.
