  Context: Performance metrics are a core component of the evaluation of any
machine learning model and used to compare models and estimate their
usefulness. Recent work started to question the validity of many performance
metrics for this purpose in the context of software defect prediction.
  Objective: Within this study, we explore the relationship between performance
metrics and the cost saving potential of defect prediction models. We study
whether performance metrics are suitable proxies to evaluate the cost saving
capabilities and derive a theory for the relationship between performance
metrics and cost saving potential.
  Methods: We measure performance metrics and cost saving potential in defect
prediction experiments. We use a multinomial logit model, decision, and random
forest to model the relationship between the metrics and the cost savings.
  Results: We could not find a stable relationship between cost savings and
performance metrics. We attribute the lack of the relationship to the inability
of performance metrics to account for the property that a small proportion of
very large software artifacts are the main driver of the costs.
  Conclusion: Any defect prediction study interested in finding the best
prediction model, must consider cost savings directly, because no reasonable
claims regarding the economic benefits of defect prediction can be made
otherwise.
