  Metrics and frameworks to quantifiably assess security measures have arisen
from needs of three distinct research communities - statistical measures from
the intrusion detection and prevention literature, evaluation of cyber
exercises, e.g.,red-team and capture-the-flag competitions, and economic
analyses addressing cost-versus-security tradeoffs. In this paper we provide
two primary contributions to the security evaluation literature - a
representative survey, and a novel framework for evaluating security that is
flexible, applicable to all three use cases, and readily interpretable. In our
survey of the literature we identify the distinct themes from each community's
evaluation procedures side by side and flesh out the drawbacks and benefits of
each. The evaluation framework we propose includes comprehensively modeling the
resource, labor, and attack costs in dollars incurred based on expected
resource usage, accuracy metrics, and time. This framework provides a unified
approach in that it incorporates the accuracy and performance metrics, which
dominate intrusion detection evaluation, the time to detection and impact to
data and resources of an attack, favored by educational competitions' metrics,
and the monetary cost of many essential security components used in financial
analysis. Moreover, it is flexible enough to accommodate each use case, easily
interpretable and comparable, and comprehensive in terms of costs
considered.Finally, we provide two examples of the framework applied to
real-world use cases. Overall, we provide a survey and a grounded, flexible
framework with multiple concrete examples for evaluating security which can
address the needs of three currently distinct communities.
