  We analyze the claims that video recreations of shoulder surfing attacks
offer a suitable alternative and a baseline, as compared to evaluation in a
live setting. We recreated a subset of the factors of a prior video-simulation
experiment conducted by Aviv et al. (ACSAC 2017), and model the same scenario
using live participants ($n=36$) instead (i.e., the victim and attacker were
both present). The live experiment confirmed that for Android's graphical
patterns video simulation is consistent with the live setting for attacker
success rates. However, both 4- and 6-digit PINs demonstrate statistically
significant differences in attacker performance, with live attackers performing
as much 1.9x better than in the video simulation. The security benefits gained
from removing feedback lines in Android's graphical patterns are also greatly
diminished in the live setting, particularly under multiple attacker
observations, but overall, the data suggests that video recreations can provide
a suitable baseline measure for attacker success rate. However, we caution that
researchers should consider that these baselines may greatly underestimate the
threat of an attacker in live settings.
