  Non-deterministic test behavior, or flakiness, is common and dreaded among
developers. Researchers have studied the issue and proposed approaches to
mitigate it. However, the vast majority of previous work has only considered
developer-written tests. The prevalence and nature of flaky tests produced by
test generation tools remain largely unknown. We ask whether such tools also
produce flaky tests and how these differ from developer-written ones.
Furthermore, we evaluate mechanisms that suppress flaky test generation. We
sample 6 356 projects written in Java or Python. For each project, we generate
tests using EvoSuite (Java) and Pynguin (Python), and execute each test 200
times, looking for inconsistent outcomes. Our results show that flakiness is at
least as common in generated tests as in developer-written tests. Nevertheless,
existing flakiness suppression mechanisms implemented in EvoSuite are effective
in alleviating this issue (71.7 % fewer flaky tests). Compared to
developer-written flaky tests, the causes of generated flaky tests are
distributed differently. Their non-deterministic behavior is more frequently
caused by randomness, rather than by networking and concurrency. Using
flakiness suppression, the remaining flaky tests differ significantly from any
flakiness previously reported, where most are attributable to runtime
optimizations and EvoSuite-internal resource thresholds. These insights, with
the accompanying dataset, can help maintainers to improve test generation
tools, give recommendations for developers using these tools, and serve as a
foundation for future research in test flakiness or test generation.
