  The cost of errors related to machine learning classifiers, namely, false
positives and false negatives, are not equal and are application dependent. For
example, in cybersecurity applications, the cost of not detecting an attack is
very different from marking a benign activity as an attack. Various design
choices during machine learning model building, such as hyperparameter tuning
and model selection, allow a data scientist to trade-off between these two
errors. However, most of the commonly used metrics to evaluate model quality,
such as $F_1$ score, which is defined in terms of model precision and recall,
treat both these errors equally, making it difficult for users to optimize for
the actual cost of these errors. In this paper, we propose a new cost-aware
metric, $C_{score}$ based on precision and recall that can replace $F_1$ score
for model evaluation and selection. It includes a cost ratio that takes into
account the differing costs of handling false positives and false negatives. We
derive and characterize the new cost metric, and compare it to $F_1$ score.
Further, we use this metric for model thresholding for five cybersecurity
related datasets for multiple cost ratios. The results show an average cost
savings of 49%.
