  Explainable Artificial Intelligence (XAI) aims to improve the transparency of
machine learning (ML) pipelines. We systematize the increasingly growing (but
fragmented) microcosm of studies that develop and utilize XAI methods for
defensive and offensive cybersecurity tasks. We identify 3 cybersecurity
stakeholders, i.e., model users, designers, and adversaries, who utilize XAI
for 4 distinct objectives within an ML pipeline, namely 1) XAI-enabled user
assistance, 2) XAI-enabled model verification, 3) explanation verification &
robustness, and 4) offensive use of explanations. Our analysis of the
literature indicates that many of the XAI applications are designed with little
understanding of how they might be integrated into analyst workflows -- user
studies for explanation evaluation are conducted in only 14% of the cases. The
security literature sometimes also fails to disentangle the role of the various
stakeholders, e.g., by providing explanations to model users and designers
while also exposing them to adversaries. Additionally, the role of model
designers is particularly minimized in the security literature. To this end, we
present an illustrative tutorial for model designers, demonstrating how XAI can
help with model verification. We also discuss scenarios where interpretability
by design may be a better alternative. The systematization and the tutorial
enable us to challenge several assumptions, and present open problems that can
help shape the future of XAI research within cybersecurity.
