  Diff-based mutation testing is a mutation testing approach that only mutates
lines affected by a code change under review. Google's mutation testing service
integrates diff-based mutation testing into the code review process and
continuously gathers developer feedback on mutants surfaced during code review.
To enhance the developer experience, the mutation testing service implements a
number of suppression rules, which target not-useful mutants-that is, mutants
that have consistently received negative developer feedback. However, while
effective, manually implementing suppression rules require significant
engineering time. An automatic system to rank and suppress mutants would
facilitate the maintenance of the mutation testing service. This paper proposes
and evaluates MuRS, an automated approach that groups mutants by patterns in
the source code under test and uses these patterns to rank and suppress future
mutants based on historical developer feedback on mutants in the same group. To
evaluate MuRS, we conducted an A/B testing study, comparing MuRS to the
existing mutation testing service. Despite the strong baseline, which uses
manually developed suppression rules, the results show a statistically
significantly lower negative feedback ratio of 11.45% for MuRS versus 12.41%
for the baseline. The results also show that MuRS is able to recover existing
suppression rules implemented in the baseline. Finally, the results show that
statement-deletion mutant groups received both the most positive and negative
developer feedback, suggesting a need for additional context that can
distinguish between useful and not-useful mutants in these groups. Overall,
MuRS has the potential to substantially reduce the development and maintenance
cost for an effective mutation testing service by automatically learning
suppression rules.
