  In this paper we introduce SAIBERSOC, a tool and methodology enabling
security researchers and operators to evaluate the performance of deployed and
operational Security Operation Centers (SOCs) (or any other security monitoring
infrastructure). The methodology relies on the MITRE ATT&CK Framework to define
a procedure to generate and automatically inject synthetic attacks in an
operational SOC to evaluate any output metric of interest (e.g., detection
accuracy, time-to-investigation, etc.). To evaluate the effectiveness of the
proposed methodology, we devise an experiment with $n=124$ students playing the
role of SOC analysts. The experiment relies on a real SOC infrastructure and
assigns students to either a BADSOC or a GOODSOC experimental condition. Our
results show that the proposed methodology is effective in identifying
variations in SOC performance caused by (minimal) changes in SOC configuration.
We release the SAIBERSOC tool implementation as free and open source software.
