  Mutation testing has been demonstrated to be one of the most powerful
fault-revealing tools in the tester's tool kit. Much previous work implicitly
assumed it to be sufficient to re-compute mutant suites per release. Sadly,
this makes mutation results inconsistent; mutant scores from each release
cannot be directly compared, making it harder to measure test improvement.
Furthermore, regular code change means that a mutant suite's relevance will
naturally degrade over time. We measure this degradation in relevance for
143,500 mutants in 4 non-trivial systems finding that, on overage, 52% degrade.
We introduce a mutant brittleness measure and use it to audit software systems
and their mutation suites. We also demonstrate how consistent-by-construction
long-standing mutant suites can be identified with a 10x improvement in mutant
relevance over an arbitrary test suite. Our results indicate that the research
community should avoid the re-computation of mutant suites and focus, instead,
on long-standing mutants, thereby improving the consistency and relevance of
mutation testing.
