  While neural network binary classifiers are often evaluated on metrics such
as Accuracy and $F_1$-Score, they are commonly trained with a cross-entropy
objective. How can this training-evaluation gap be addressed? While specific
techniques have been adopted to optimize certain confusion matrix based
metrics, it is challenging or impossible in some cases to generalize the
techniques to other metrics. Adversarial learning approaches have also been
proposed to optimize networks via confusion matrix based metrics, but they tend
to be much slower than common training methods. In this work, we propose a
unifying approach to training neural network binary classifiers that combines a
differentiable approximation of the Heaviside function with a probabilistic
view of the typical confusion matrix values using soft sets. Our theoretical
analysis shows the benefit of using our method to optimize for a given
evaluation metric, such as $F_1$-Score, with soft sets, and our extensive
experiments show the effectiveness of our approach in several domains.
