  There has been a significant increase in the development of data-driven
safety analytics approaches in recent years. In light of these advances it has
become imperative to evaluate such approaches in a principled way to determine
their merits and limitations. To that end, we propose an evaluation methodology
underpinned by a simulated environment that allows for a comprehensive
assessment of safety analytics approaches. While assessing those approaches
with historical field data is undoubtedly important, such an assessment has
limited statistical power because it corresponds to only one realization. The
proposed methodology enables validation over a large number of realizations,
thereby circumventing the statistical limitations of evaluation on historical
data. Moreover, by using a simulated environment one is able to clearly
distinguish between the variability in the observed data and differences in
performance between safety approaches. A simulated environment does this by
comparing the approaches under controlled circumstances, resulting in a fair
and systematic evaluation of the potential long-term benefits. We demonstrate
the utility of the proposed methodology via a case study that compares a few
candidate safety analytics approaches. These approaches differ in how they
assimilate field safety data to assess safety risk and suggest mitigative
actions. We show that the proposed methodology indeed reveals useful insights
and quantifies the relative merits and drawbacks of the different approaches,
which would be otherwise difficult to objectively determine in a real-world
scenario.
