  While the existence of many security elements can be measured (e.g.,
vulnerabilities, security controls, or privacy controls), it is challenging to
measure their relative security impact. In the physical world we can often
measure the impact of individual elements to a system. However, in cyber
security we often lack ground truth (i.e., the ability to directly measure
significance). In this work we propose to solve this by leveraging human expert
opinion to provide ground truth. Experts are iteratively asked to compare pairs
of security elements to determine their relative significance. On the back end
our knowledge encoding tool performs a form of binary insertion sort on a set
of security elements using each expert as an oracle for the element
comparisons. The tool not only sorts the elements (note that equality may be
permitted), but it also records the strength or degree of each relationship.
The output is a directed acyclic 'constraint' graph that provides a total
ordering among the sets of equivalent elements. Multiple constraint graphs are
then unified together to form a single graph that is used to generate a scoring
or prioritization system. For our empirical study, we apply this
domain-agnostic measurement approach to generate scoring/prioritization systems
in the areas of vulnerability scoring, privacy control prioritization, and
cyber security control evaluation.
