  Information security awareness (ISA) is a practice focused on the set of
skills, which help a user successfully mitigate a social engineering attack.
Previous studies have presented various methods for evaluating the ISA of both
PC and mobile users. These methods rely primarily on subjective data sources
such as interviews, surveys, and questionnaires that are influenced by human
interpretation and sincerity. Furthermore, previous methods for evaluating ISA
did not address the differences between classes of social engineering attacks.
In this paper, we present a novel framework designed for evaluating the ISA of
smartphone users to specific social engineering attack classes. In addition to
questionnaires, the proposed framework utilizes objective data sources: a
mobile agent and a network traffic monitor; both of which are used to analyze
the actual behavior of users. We empirically evaluated the ISA scores assessed
from the three data sources (namely, the questionnaires, mobile agent, and
network traffic monitor) by conducting a long-term user study involving 162
smartphone users. All participants were exposed to four different security
challenges that resemble real-life social engineering attacks. These challenges
were used to assess the ability of the proposed framework to derive a relevant
ISA score. The results of our experiment show that: (1) the self-reported
behavior of the users differs significantly from their actual behavior; and (2)
ISA scores derived from data collected by the mobile agent or the network
traffic monitor are highly correlated with the users' success in mitigating
social engineering attacks.
