  A vision-based keystroke inference attack is a side-channel attack in which
an attacker uses an optical device to record users on their mobile devices and
infer their keystrokes. The threat space for these attacks has been studied in
the past, but we argue that the defining characteristics for this threat space,
namely the strength of the attacker, are outdated. Previous works do not study
adversaries with vision systems that have been trained with deep neural
networks because these models require large amounts of training data and
curating such a dataset is expensive. To address this, we create a large-scale
synthetic dataset to simulate the attack scenario for a keystroke inference
attack. We show that first pre-training on synthetic data, followed by adopting
transfer learning techniques on real-life data, increases the performance of
our deep learning models. This indicates that these models are able to learn
rich, meaningful representations from our synthetic data and that training on
the synthetic data can help overcome the issue of having small, real-life
datasets for vision-based key stroke inference attacks. For this work, we focus
on single keypress classification where the input is a frame of a keypress and
the output is a predicted key. We are able to get an accuracy of 95.6% after
pre-training a CNN on our synthetic data and training on a small set of
real-life data in an adversarial domain adaptation framework. Source Code for
Simulator:
https://github.com/jlim13/keystroke-inference-attack-synthetic-dataset-generator-
