  Traditionally, mutation testing generates an abundance of small deviations of
a program, called mutants. At industrial systems the scale and size of
Facebook's, doing this is infeasible. We should not create mutants that the
test suite would likely fail on or that give no actionable signal to
developers. To tackle this problem, in this paper, we semi-automatically learn
error-inducing patterns from a corpus of common Java coding errors and from
changes that caused operational anomalies at Facebook specifically. We combine
the mutations with instrumentation that measures which tests exactly visited
the mutated piece of code. Results on more than 15,000 generated mutants show
that more than half of the generated mutants survive Facebook's rigorous test
suite of unit, integration, and system tests. Moreover, in a case study with 26
developers, all but two found information of automatically detected test holes
interesting in principle. As such, almost half of the 26 would actually act on
the mutant presented to them by adapting an existing or creating a new test.
The others did not for a variety of reasons often outside the scope of mutation
testing. It remains a practical challenge how we can include such external
information to increase the true actionability rate on mutants.
