  Very few social media studies have been done on South African user-generated
content during the COVID-19 pandemic and even fewer using hand-labelling over
automated methods. Vaccination is a major tool in the fight against the
pandemic, but vaccine hesitancy jeopardizes any public health effort. In this
study, sentiment analysis on South African tweets related to vaccine hesitancy
was performed, with the aim of training AI-mediated classification models and
assessing their reliability in categorizing UGC. A dataset of 30000 tweets from
South Africa were extracted and hand-labelled into one of three sentiment
classes: positive, negative, neutral. The machine learning models used were
LSTM, bi-LSTM, SVM, BERT-base-cased and the RoBERTa-base models, whereby their
hyperparameters were carefully chosen and tuned using the WandB platform. We
used two different approaches when we pre-processed our data for comparison:
one was semantics-based, while the other was corpus-based. The pre-processing
of the tweets in our dataset was performed using both methods, respectively.
All models were found to have low F1-scores within a range of 45$\%$-55$\%$,
except for BERT and RoBERTa which both achieved significantly better measures
with overall F1-scores of 60$\%$ and 61$\%$, respectively. Topic modelling
using an LDA was performed on the miss-classified tweets of the RoBERTa model
to gain insight on how to further improve model accuracy.
