  Self-disclosed mental health diagnoses, which serve as ground truth
annotations of mental health status in the absence of clinical measures,
underpin the conclusions behind most computational studies of mental health
language from the last decade. However, psychiatric conditions are dynamic; a
prior depression diagnosis may no longer be indicative of an individual's
mental health, either due to treatment or other mitigating factors. We ask: to
what extent are self-disclosures of mental health diagnoses actually relevant
over time? We analyze recent activity from individuals who disclosed a
depression diagnosis on social media over five years ago and, in turn, acquire
a new understanding of how presentations of mental health status on social
media manifest longitudinally. We also provide expanded evidence for the
presence of personality-related biases in datasets curated using self-disclosed
diagnoses. Our findings motivate three practical recommendations for improving
mental health datasets curated using self-disclosed diagnoses: 1) Annotate
diagnosis dates and psychiatric comorbidities; 2) Sample control groups using
propensity score matching; 3) Identify and remove spurious correlations
introduced by selection bias.
