  Software failures inform engineering work, standards, regulations. For
example, the Log4J vulnerability brought government and industry attention to
evaluating and securing software supply chains. Accessing private engineering
records is difficult, so failure analyses tend to use information reported by
the news media. However, prior works in this direction have relied on manual
analysis. That has limited the scale of their analyses. The community lacks
automated support to enable such analyses to consider a wide range of news
sources and incidents.
  In this paper, we propose the Failure Analysis Investigation with LLMs (FAIL)
system to fill this gap. FAIL collects, analyzes, and summarizes software
failures as reported in the news. FAIL groups articles that describe the same
incidents. It then analyzes incidents using existing taxonomies for
postmortems, faults, and system characteristics. To tune and evaluate FAIL, we
followed the methods of prior works by manually analyzing 31 software failures.
FAIL achieved an F1 score of 90% for collecting news about software failures, a
V-measure of 0.98 for merging articles reporting on the same incident, and
extracted 90% of the facts about failures. We then applied FAIL to a total of
137,427 news articles from 11 providers published between 2010 and 2022. FAIL
identified and analyzed 2457 distinct failures reported across 4,184 articles.
Our findings include: (1) current generation of large language models are
capable of identifying news articles that describe failures, and analyzing them
according to structured taxonomies; (2) high recurrences of similar failures
within organizations and across organizations; and (3) severity of the
consequences of software failures have increased over the past decade. The full
FAIL database is available so that researchers, engineers, and policymakers can
learn from a diversity of software failures.
