  This study introduces novel superior scoring rules called Penalized Brier
Score (PBS) and Penalized Logarithmic Loss (PLL) to improve model evaluation
for probabilistic classification. Traditional scoring rules like Brier Score
and Logarithmic Loss sometimes assign better scores to misclassifications in
comparison with correct classifications. This discrepancy from the actual
preference for rewarding correct classifications can lead to suboptimal model
selection. By integrating penalties for misclassifications, PBS and PLL modify
traditional proper scoring rules to consistently assign better scores to
correct predictions. Formal proofs demonstrate that PBS and PLL satisfy
strictly proper scoring rule properties while also preferentially rewarding
accurate classifications. Experiments showcase the benefits of using PBS and
PLL for model selection, model checkpointing, and early stopping. PBS exhibits
a higher negative correlation with the F1 score compared to the Brier Score
during training. Thus, PBS more effectively identifies optimal checkpoints and
early stopping points, leading to improved F1 scores. Comparative analysis
verifies models selected by PBS and PLL achieve superior F1 scores. Therefore,
PBS and PLL address the gap between uncertainty quantification and accuracy
maximization by encapsulating both proper scoring principles and explicit
preference for true classifications. The proposed metrics can enhance model
evaluation and selection for reliable probabilistic classification.
