  Modern security operation centers (SOCs) rely on operators and a tapestry of
logging and alerting tools with large scale collection and query abilities. SOC
investigations are tedious as they rely on manual efforts to query diverse data
sources, overlay related logs, and correlate the data into information and then
document results in a ticketing system. Security orchestration, automation, and
response (SOAR) tools are a new technology that promise to collect, filter, and
display needed data; automate common tasks that require SOC analysts' time;
facilitate SOC collaboration; and, improve both efficiency and consistency of
SOCs. SOAR tools have never been tested in practice to evaluate their effect
and understand them in use. In this paper, we design and administer the first
hands-on user study of SOAR tools, involving 24 participants and 6 commercial
SOAR tools. Our contributions include the experimental design, itemizing six
characteristics of SOAR tools and a methodology for testing them. We describe
configuration of the test environment in a cyber range, including network,
user, and threat emulation; a full SOC tool suite; and creation of artifacts
allowing multiple representative investigation scenarios to permit testing. We
present the first research results on SOAR tools. We found that SOAR
configuration is critical, as it involves creative design for data display and
automation. We found that SOAR tools increased efficiency and reduced context
switching during investigations, although ticket accuracy and completeness
(indicating investigation quality) decreased with SOAR use. Our findings
indicated that user preferences are slightly negatively correlated with their
performance with the tool; overautomation was a concern of senior analysts, and
SOAR tools that balanced automation with assisting a user to make decisions
were preferred.
