  Traditional evaluation metrics for classification in natural language
processing such as accuracy and area under the curve fail to differentiate
between models with different predictive behaviors despite their similar
performance metrics. We introduce sensitivity score, a metric that scrutinizes
models' behaviors at the vocabulary level to provide insights into disparities
in their decision-making logic. We assess the sensitivity score on a set of
representative words in the test set using two classifiers trained for hospital
readmission classification with similar performance statistics. Our experiments
compare the decision-making logic of clinicians and classifiers based on rank
correlations of sensitivity scores. The results indicate that the language
model's sensitivity score aligns better with the professionals than the xgboost
classifier on tf-idf embeddings, which suggests that xgboost uses some spurious
features. Overall, this metric offers a novel perspective on assessing models'
robustness by quantifying their discrepancy with professional opinions. Our
code is available on GitHub (https://github.com/nyuolab/Model_Sensitivity).
