  In this paper, we define visual log of a software system as data capturing
the interactions between its users and its graphic user interface (GUI), such
as screen-shots and screen recordings. We vision that mining such visual log
could be useful for bug reproducing and debugging, automated GUI testing, user
interface designing, question answering of common usages in software support,
etc. Toward that vision, we propose a core framework for mining visual log of
software. This framework focuses on detecting GUI elements and changes in
visual log, removing users' private data, recognizing user interactions with
GUI elements, and learning GUI usage patterns. We also performed a small study
on the characteristics of GUI elements in mobile apps. The findings from this
study suggested several heuristics to design techniques for recognizing GUI
elements and interactions.
