  Time roots in applying language models for biomedical applications: models
are trained on historical data and will be deployed for new or future data,
which may vary from training data. While increasing biomedical tasks have
employed state-of-the-art language models, there are very few studies have
examined temporal effects on biomedical models when data usually shifts across
development and deployment. This study fills the gap by statistically probing
relations between language model performance and data shifts across three
biomedical tasks. We deploy diverse metrics to evaluate model performance,
distance methods to measure data drifts, and statistical methods to quantify
temporal effects on biomedical language models. Our study shows that time
matters for deploying biomedical language models, while the degree of
performance degradation varies by biomedical tasks and statistical
quantification approaches. We believe this study can establish a solid
benchmark to evaluate and assess temporal effects on deploying biomedical
language models.
