  Validation of Android apps via testing is difficult owing to the presence of
flaky tests. Due to non-deterministic execution environments, a sequence of
events (a test) may lead to success or failure in unpredictable ways. In this
work, we present an approach and tool FlakeShovel for detecting flaky tests
through systematic exploration of event orders. Our key observation is that for
a test in a mobile app, there is a testing framework thread which creates the
test events, a main User-Interface (UI) thread processing these events, and
there may be several other background threads running asynchronously. For any
event e whose execution involves potential non-determinism, we localize the
earliest (latest) event after (before) which e must happen.We then efficiently
explore the schedules between the upper/lower bound events while grouping
events within a single statement, to find whether the test outcome is flaky. We
also create a suite of subject programs called DroidFlaker to study flaky tests
in Android apps. Our experiments on subject-suite DroidFlaker demonstrate the
efficacy of our flaky test detection. Our work is complementary to existing
flaky test detection tools like Deflaker which check only failing tests.
FlakeShovel can detect flaky tests among passing tests, as shown by our
approach and experiments.
