  Previous studies have highlighted the importance of vaccination as an
effective strategy to control the transmission of the COVID-19 virus. It is
crucial for policymakers to have a comprehensive understanding of the public's
stance towards vaccination on a large scale. However, attitudes towards
COVID-19 vaccination, such as pro-vaccine or vaccine hesitancy, have evolved
over time on social media. Thus, it is necessary to account for possible
temporal shifts when analysing these stances. This study aims to examine the
impact of temporal concept drift on stance detection towards COVID-19
vaccination on Twitter. To this end, we evaluate a range of transformer-based
models using chronological (splitting the training, validation, and test sets
in order of time) and random splits (randomly splitting these three sets) of
social media data. Our findings reveal significant discrepancies in model
performance between random and chronological splits in several existing
COVID-19-related datasets; specifically, chronological splits significantly
reduce the accuracy of stance classification. Therefore, real-world stance
detection approaches need to be further refined to incorporate temporal factors
as a key consideration.
