  Transformer-based machine learning models have become an essential tool for
many natural language processing (NLP) tasks since the introduction of the
method. A common objective of these projects is to classify text data.
Classification models are often extended to a different topic and/or time
period. In these situations, deciding how long a classification is suitable for
and when it is worth re-training our model is difficult. This paper compares
different approaches to fine-tune a BERT model for a long-running
classification task. We use data from different periods to fine-tune our
original BERT model, and we also measure how a second round of annotation could
boost the classification quality. Our corpus contains over 8 million comments
on COVID-19 vaccination in Hungary posted between September 2020 and December
2021. Our results show that the best solution is using all available unlabeled
comments to fine-tune a model. It is not advisable to focus only on comments
containing words that our model has not encountered before; a more efficient
solution is randomly sample comments from the new period. Fine-tuning does not
prevent the model from losing performance but merely slows it down. In a
rapidly changing linguistic environment, it is not possible to maintain model
performance without regularly annotating new text.
