  We present and evaluate Spectrum-Based Log Diagnosis (SBLD), a method to help
developers quickly diagnose problems found in complex integration and
deployment runs. Inspired by Spectrum-Based Fault Localization, SBLD leverages
the differences in event occurrences between logs for failing and passing runs,
to highlight events that are stronger associated with failing runs.
  Using data provided by our industrial partner, we empirically investigate the
following questions: (i) How well does SBLD reduce the effort needed to
identify all failure-relevant events in the log for a failing run? (ii) How is
the performance of SBLD affected by available data? (iii) How does SBLD compare
to searching for simple textual patterns that often occur in failure-relevant
events? We answer (i) and (ii) using summary statistics and heatmap
visualizations, and for (iii) we compare three configurations of SBLD (with
resp. minimum, median and maximum data) against a textual search using Wilcoxon
signed-rank tests and the Vargha-Delaney measure of stochastic superiority.
  Our evaluation shows that (i) SBLD achieves a significant effort reduction
for the dataset used, (ii) SBLD benefits from additional logs for passing runs
in general, and it benefits from additional logs for failing runs when there is
a proportional amount of logs for passing runs in the data. Finally, (iii) SBLD
and textual search are roughly equally effective at effort-reduction, while
textual search has a slightly better recall. We investigate the cause, and
discuss how it is due to the characteristics of a specific part of our data.
  We conclude that SBLD shows promise as a method for diagnosing failing runs,
that its performance is positively affected by additional data, but that it
does not outperform textual search on the dataset considered. Future work
includes investigating SBLD's generalizability on additional datasets.
