  Models that accurately detect depression from text are important tools for
addressing the post-pandemic mental health crisis. BERT-based classifiers'
promising performance and the off-the-shelf availability make them great
candidates for this task. However, these models are known to suffer from
performance inconsistencies and poor generalization. In this paper, we
introduce the DECK (DEpression ChecKlist), depression-specific model
behavioural tests that allow better interpretability and improve
generalizability of BERT classifiers in depression domain. We create 23 tests
to evaluate BERT, RoBERTa and ALBERT depression classifiers on three datasets,
two Twitter-based and one clinical interview-based. Our evaluation shows that
these models: 1) are robust to certain gender-sensitive variations in text; 2)
rely on the important depressive language marker of the increased use of first
person pronouns; 3) fail to detect some other depression symptoms like suicidal
ideation. We also demonstrate that DECK tests can be used to incorporate
symptom-specific information in the training data and consistently improve
generalizability of all three BERT models, with an out-of-distribution F1-score
increase of up to 53.93%.
