  Social media allows researchers to track societal and cultural changes over
time based on language analysis tools. Many of these tools rely on statistical
algorithms which need to be tuned to specific types of language. Recent studies
have shown the absence of appropriate tuning, specifically in the presence of
semantic shift, can hinder robustness of the underlying methods. However,
little is known about the practical effect this sensitivity may have on
downstream longitudinal analyses. We explore this gap in the literature through
a timely case study: understanding shifts in depression during the course of
the COVID-19 pandemic. We find that inclusion of only a small number of
semantically-unstable features can promote significant changes in longitudinal
estimates of our target outcome. At the same time, we demonstrate that a
recently-introduced method for measuring semantic shift may be used to
proactively identify failure points of language-based models and, in turn,
improve predictive generalization.
