  The advent of transformer-based architectures and large language models
(LLMs) have significantly advanced the performance of natural language
processing (NLP) models. Since these LLMs are trained on huge corpuses of data
from the web and other sources, there has been a major concern about harmful
prejudices that may potentially be transferred from the data. In many
applications, these pre-trained LLMs are fine-tuned on task specific datasets,
which can further contribute to biases. This paper studies the extent of biases
absorbed by LLMs during pre-training as well as task-specific behaviour after
fine-tuning. We found that controlled interventions on pre-trained LLMs, prior
to fine-tuning, have minimal effect on lowering biases in classifiers. However,
the biases present in domain-specific datasets play a much bigger role, and
hence mitigating them at this stage has a bigger impact. While pre-training
does matter, but after the model has been pre-trained, even slight changes to
co-occurrence rates in the fine-tuning dataset has a significant effect on the
bias of the model.
