  Large pre-trained language models are widely used in the community. These
models are usually trained on unmoderated and unfiltered data from open sources
like the Internet. Due to this, biases that we see in platforms online which
are a reflection of those in society are in turn captured and learned by these
models. These models are deployed in applications that affect millions of
people and their inherent biases are harmful to the targeted social groups. In
this work, we study the general trend in bias reduction as newer pre-trained
models are released. Three recent models ( ELECTRA, DeBERTa, and DistilBERT)
are chosen and evaluated against two bias benchmarks, StereoSet and
CrowS-Pairs. They are compared to the baseline of BERT using the associated
metrics. We explore whether as advancements are made and newer, faster, lighter
models are released: are they being developed responsibly such that their
inherent social biases have been reduced compared to their older counterparts?
The results are compiled and we find that all the models under study do exhibit
biases but have generally improved as compared to BERT.
