  Blind people frequently encounter inaccessible dynamic touchscreens in their
everyday lives that are difficult, frustrating, and often impossible to use
independently. Touchscreens are often the only way to control everything from
coffee machines and payment terminals, to subway ticket machines and in-flight
entertainment systems. Interacting with dynamic touchscreens is difficult
non-visually because the visual user interfaces change, interactions often
occur over multiple different screens, and it is easy to accidentally trigger
interface actions while exploring the screen. To solve these problems, we
introduce StateLens - a three-part reverse engineering solution that makes
existing dynamic touchscreens accessible. First, StateLens reverse engineers
the underlying state diagrams of existing interfaces using point-of-view videos
found online or taken by users using a hybrid crowd-computer vision pipeline.
Second, using the state diagrams, StateLens automatically generates
conversational agents to guide blind users through specifying the tasks that
the interface can perform, allowing the StateLens iOS application to provide
interactive guidance and feedback so that blind users can access the interface.
Finally, a set of 3D-printed accessories enable blind people to explore
capacitive touchscreens without the risk of triggering accidental touches on
the interface. Our technical evaluation shows that StateLens can accurately
reconstruct interfaces from stationary, hand-held, and web videos; and, a user
study of the complete system demonstrates that StateLens successfully enables
blind users to access otherwise inaccessible dynamic touchscreens.
