  Automated understanding of user interfaces (UIs) from their pixels can
improve accessibility, enable task automation, and facilitate interface design
without relying on developers to comprehensively provide metadata. A first step
is to infer what UI elements exist on a screen, but current approaches are
limited in how they infer how those elements are semantically grouped into
structured interface definitions. In this paper, we motivate the problem of
screen parsing, the task of predicting UI elements and their relationships from
a screenshot. We describe our implementation of screen parsing and provide an
effective training procedure that optimizes its performance. In an evaluation
comparing the accuracy of the generated output, we find that our implementation
significantly outperforms current systems (up to 23%). Finally, we show three
example applications that are facilitated by screen parsing: (i) UI similarity
search, (ii) accessibility enhancement, and (iii) code generation from UI
screenshots.
