  Flaky tests are software tests that exhibit a seemingly random outcome (pass
or fail) when run against the same, identical code. Previous work has examined
fixes to flaky tests and has proposed automated solutions to locate as well as
fix flaky tests--we complement it by examining the perceptions of software
developers about the nature, relevance, and challenges of this phenomenon.
  We asked 21 professional developers to classify 200 flaky tests they
previously fixed, in terms of the nature of the flakiness, the origin of the
flakiness, and the fixing effort. We complement this analysis with information
about the fixing strategy. Subsequently, we conducted an online survey with 121
developers with a median industrial programming experience of five years. Our
research shows that: The flakiness is due to several different causes, four of
which have never been reported before, despite being the most costly to fix;
flakiness is perceived as significant by the vast majority of developers,
regardless of their team's size and project's domain, and it can have effects
on resource allocation, scheduling, and the perceived reliability of the test
suite; and the challenges developers report to face regard mostly the
reproduction of the flaky behavior and the identification of the cause for the
flakiness. Data and materials [https://doi.org/10.5281/zenodo.3265785].
