  Security operation centers (SOCs) all over the world are tasked with reacting
to cybersecurity alerts ranging in severity. Security Orchestration,
Automation, and Response (SOAR) tools streamline cybersecurity alert responses
by SOC operators. SOAR tool adoption is expensive both in effort and finances.
Hence, it is crucial to limit adoption to those most worthwhile; yet no
research evaluating or comparing SOAR tools exists. The goal of this work is to
evaluate several SOAR tools using specific criteria pertaining to their
usability. SOC operators were asked to first complete a survey about what SOAR
tool aspects are most important. Operators were then assigned a set of SOAR
tools for which they viewed demonstration and overview videos, and then
operators completed a second survey wherein they were tasked with evaluating
each of the tools on the aspects from the first survey. In addition, operators
provided an overall rating to each of their assigned tools, and provided a
ranking of their tools in order of preference. Due to time constraints on SOC
operators for thorough testing, we provide a systematic method of downselecting
a large pool of SOAR tools to a select few that merit next-step hands-on
evaluation by SOC operators. Furthermore, the analyses conducted in this survey
help to inform future development of SOAR tools to ensure that the appropriate
functions are available for use in a SOC.
