  Flaky tests (tests with non-deterministic outcomes) pose a major challenge
for software testing. They are known to cause significant issues such as
reducing the effectiveness and efficiency of testing and delaying software
releases. In recent years, there has been an increased interest in flaky tests,
with research focusing on different aspects of flakiness, such as identifying
causes, detection methods and mitigation strategies. Test flakiness has also
become a key discussion point for practitioners (in blog posts, technical
magazines, etc.) as the impact of flaky tests is felt across the industry. This
paper presents a multivocal review that investigates how flaky tests, as a
topic, have been addressed in both research and practice. We cover a total of
651 articles (560 academic articles and 91 grey literature articles/posts), and
structure the body of relevant research and knowledge using four different
dimensions: causes, detection, impact and responses. For each of those
dimensions we provide a categorisation, and classify existing research,
discussions, methods and tools. With this, we provide a comprehensive and
current snapshot of existing thinking on test flakiness, covering both academic
views and industrial practices, and identify limitations and opportunities for
future research.
