  Reviewing source code from a security perspective has proven to be a
difficult task. Indeed, previous research has shown that developers often miss
even popular and easy-to-detect vulnerabilities during code review. Initial
evidence suggests that a significant cause may lie in the reviewers' mental
attitude and common practices. In this study, we investigate whether and how
explicitly asking developers to focus on security during a code review affects
the detection of vulnerabilities. Furthermore, we evaluate the effect of
providing a security checklist to guide the security review. To this aim, we
conduct an online experiment with 150 participants, of which 71% report to have
three or more years of professional development experience. Our results show
that simply asking reviewers to focus on security during the code review
increases eight times the probability of vulnerability detection. The presence
of a security checklist does not significantly improve the outcome further,
even when the checklist is tailored to the change under review and the existing
vulnerabilities in the change. These results provide evidence supporting the
mental attitude hypothesis and call for further work on security checklists'
effectiveness and design. Data and materials:
https://doi.org/10.5281/zenodo.6026291
