  Mobile User Interface Summarization generates succinct language descriptions
of mobile screens for conveying important contents and functionalities of the
screen, which can be useful for many language-based application scenarios. We
present Screen2Words, a novel screen summarization approach that automatically
encapsulates essential information of a UI screen into a coherent language
phrase. Summarizing mobile screens requires a holistic understanding of the
multi-modal data of mobile UIs, including text, image, structures as well as UI
semantics, motivating our multi-modal learning approach. We collected and
analyzed a large-scale screen summarization dataset annotated by human workers.
Our dataset contains more than 112k language summarization across $\sim$22k
unique UI screens. We then experimented with a set of deep models with
different configurations. Our evaluation of these models with both automatic
accuracy metrics and human rating shows that our approach can generate
high-quality summaries for mobile screens. We demonstrate potential use cases
of Screen2Words and open-source our dataset and model to lay the foundations
for further bridging language and user interfaces.
