  The task of designing secure software systems is fraught with uncertainty, as
data on uncommon attacks is limited, costs are difficult to estimate, and
technology and tools are continually changing. Consequently, experts may
interpret the security risks posed to a system in different ways, leading to
variation in assessment. This paper presents research into measuring the
variability in decision making between security professionals, with the
ultimate goal of improving the quality of security advice given to software
system designers. A set of thirty nine cyber-security experts took part in an
exercise in which they independently assessed a realistic system scenario. This
study quantifies agreement in the opinions of experts, examines methods of
aggregating opinions, and produces an assessment of attacks from ratings of
their components. We show that when aggregated, a coherent consensus view of
security emerges which can be used to inform decisions made during systems
design.
