  Context: There is considerable diversity in the range and design of
computational experiments to assess classifiers for software defect prediction.
This is particularly so, regarding the choice of classifier performance
metrics. Unfortunately some widely used metrics are known to be biased, in
particular F1. Objective: We want to understand the extent to which the
widespread use of the F1 renders empirical results in software defect
prediction unreliable. Method: We searched for defect prediction studies that
report both F1 and the Matthews correlation coefficient (MCC). This enabled us
to determine the proportion of results that are consistent between both metrics
and the proportion that change. Results: Our systematic review identifies 8
studies comprising 4017 pairwise results. Of these results, the direction of
the comparison changes in 23% of the cases when the unbiased MCC metric is
employed. Conclusion: We find compelling reasons why the choice of
classification performance metric matters, specifically the biased and
misleading F1 metric should be deprecated.
