  For large-scale simulation codes with huge and complex code bases, where
bit-for-bit comparisons are too restrictive, finding the source of
statistically significant discrepancies (e.g., from a previous version,
alternative hardware or supporting software stack) in output is non-trivial at
best. Although there are many tools for program comprehension through debugging
or slicing, few (if any) scale to a model as large as the Community Earth
System Model (CESM; trademarked), which consists of more than 1.5 million lines
of Fortran code. Currently for the CESM, we can easily determine whether a
discrepancy exists in the output using a by now well-established statistical
consistency testing tool. However, this tool provides no information as to the
possible cause of the detected discrepancy, leaving developers in a seemingly
impossible (and frustrating) situation. Therefore, our aim in this work is to
provide the tools to enable developers to trace a problem detected through the
CESM output to its source. To this end, our strategy is to reduce the search
space for the root cause(s) to a tractable size via a series of techniques that
include creating a directed graph of internal CESM variables, extracting a
subgraph (using a form of hybrid program slicing), partitioning into
communities, and ranking nodes by centrality. Runtime variable sampling then
becomes feasible in this reduced search space. We demonstrate the utility of
this process on multiple examples of CESM simulation output by illustrating how
sampling can be performed as part of an efficient parallel iterative refinement
procedure to locate error sources, including sensitivity to CPU instructions.
By providing CESM developers with tools to identify and understand the reason
for statistically distinct output, we have positively impacted the CESM
software development cycle and, in particular, its focus on quality assurance.
