  Screen recordings of mobile apps are a popular and readily available way for
users to share how they interact with apps, such as in online tutorial videos,
user reviews, or as attachments in bug reports. Unfortunately, both people and
systems can find it difficult to reproduce touch-driven interactions from video
pixel data alone. In this paper, we introduce an approach to extract and replay
user interactions in videos of mobile apps, using only pixel information in
video frames. To identify interactions, we apply heuristic-based image
processing and convolutional deep learning to segment screen recordings,
classify the interaction in each segment, and locate the interaction point. To
replay interactions on another device, we match elements on app screens using
UI element detection. We evaluate the feasibility of our pixel-based approach
using two datasets: the Rico mobile app dataset and a new dataset of 64 apps
with both iOS and Android versions. We find that our end-to-end approach can
successfully replay a majority of interactions (iOS--84.1%, Android--78.4%) on
different devices, which is a step towards supporting a variety of scenarios,
including automatically annotating interactions in existing videos, automated
UI testing, and creating interactive app tutorials.
