  In this paper, we present findings from a large-scale and long-term phishing
experiment that we conducted in collaboration with a partner company. Our
experiment ran for 15 months during which time more than 14,000 study
participants (employees of the company) received different simulated phishing
emails in their normal working context. We also deployed a reporting button to
the company's email client which allowed the participants to report suspicious
emails they received. We measured click rates for phishing emails, dangerous
actions such as submitting credentials, and reported suspicious emails.
  The results of our experiment provide three types of contributions. First,
some of our findings support previous literature with improved ecological
validity. One example of such results is good effectiveness of warnings on
emails. Second, some of our results contradict prior literature and common
industry practices. Surprisingly, we find that embedded training during
simulated phishing exercises, as commonly deployed in the industry today, does
not make employees more resilient to phishing, but instead it can have
unexpected side effects that can make employees even more susceptible to
phishing. And third, we report new findings. In particular, we are the first to
demonstrate that using the employees as a collective phishing detection
mechanism is practical in large organizations. Our results show that such
crowd-sourcing allows fast detection of new phishing campaigns, the operational
load for the organization is acceptable, and the employees remain active over
long periods of time.
