  The HTML5 <canvas> is used to display high quality graphics in web
applications such as web games (i.e., <canvas> games). However, automatically
testing <canvas> games is not possible with existing web testing techniques and
tools, and manual testing is laborious. Many widely used web testing tools rely
on the Document Object Model (DOM) to drive web test automation, but the
contents of the <canvas> are not represented in the DOM. The main alternative
approach, snapshot testing, involves comparing oracle snapshot images with
test-time snapshot images using an image similarity metric to catch visual
bugs, i.e., bugs in the graphics of the web application. However, creating and
maintaining oracle snapshot images for <canvas> games is onerous, defeating the
purpose of test automation. In this paper, we present a novel approach to
automatically detect visual bugs in <canvas> games. By leveraging an internal
representation of objects on the <canvas>, we decompose snapshot images into a
set of object images, each of which is compared with a respective oracle asset
(e.g., a sprite) using four similarity metrics: percentage overlap, mean
squared error, structural similarity, and embedding similarity. We evaluate our
approach by injecting 24 visual bugs into a custom <canvas> game, and find that
our approach achieves an accuracy of 100%, compared to an accuracy of 44.6%
with traditional snapshot testing.
