  Machine learning based phishing website detectors (ML-PWD) are a critical
part of today's anti-phishing solutions in operation. Unfortunately, ML-PWD are
prone to adversarial evasions, evidenced by both academic studies and analyses
of real-world adversarial phishing webpages. However, existing works mostly
focused on assessing adversarial phishing webpages against ML-PWD, while
neglecting a crucial aspect: investigating whether they can deceive the actual
target of phishing -- the end users. In this paper, we fill this gap by
conducting two user studies (n=470) to examine how human users perceive
adversarial phishing webpages, spanning both synthetically crafted ones (which
we create by evading a state-of-the-art ML-PWD) as well as real adversarial
webpages (taken from the wild Web) that bypassed a production-grade ML-PWD. Our
findings confirm that adversarial phishing is a threat to both users and
ML-PWD, since most adversarial phishing webpages have comparable effectiveness
on users w.r.t. unperturbed ones. However, not all adversarial perturbations
are equally effective. For example, those with added typos are significantly
more noticeable to users, who tend to overlook perturbations of higher visual
magnitude (such as replacing the background). We also show that users'
self-reported frequency of visiting a brand's website has a statistically
negative correlation with their phishing detection accuracy, which is likely
caused by overconfidence. We release our resources.
