  Non-deterministically behaving (i.e., flaky) tests hamper regression testing
as they destroy trust and waste computational and human resources. Eradicating
flakiness in test suites is therefore an important goal, but automated
debugging tools are needed to support developers when trying to understand the
causes of flakiness. A popular example for an automated approach to support
regular debugging is spectrum-based fault localization (SFL), a technique that
identifies software components that are most likely the causes of failures.
While it is possible to also apply SFL for locating likely sources of flakiness
in code, unfortunately the flakiness makes SFL both imprecise and
non-deterministic. In this paper we introduce SFFL (Spectrum-based Flaky Fault
Localization), an extension of traditional coverage-based SFL that exploits our
observation that 80% of flaky tests exhibit varying coverage behavior between
different runs. By distinguishing between stable and flaky coverage, SFFL is
able to locate the sources of flakiness more precisely and keeps the
localization itself deterministic. An evaluation on 101 flaky tests taken from
48 open-source Python projects demonstrates that SFFL is effective: Of five
prominent SFL formulas, DStar, Ochiai, and Op2 yield the best overall
performance. On average, they are able to narrow down the fault's location to
3.5 % of the project's code base, which is 18.7 % better than traditional SFL
(for DStar). SFFL's effectiveness, however, depends on the root causes of
flakiness: The source of non-order-dependent flaky tests can be located far
more precisely than order-dependent faults.
