  The pervasive influence of social biases in language data has sparked the
need for benchmark datasets that capture and evaluate these biases in Large
Language Models (LLMs). Existing efforts predominantly focus on English
language and the Western context, leaving a void for a reliable dataset that
encapsulates India's unique socio-cultural nuances. To bridge this gap, we
introduce IndiBias, a comprehensive benchmarking dataset designed specifically
for evaluating social biases in the Indian context. We filter and translate the
existing CrowS-Pairs dataset to create a benchmark dataset suited to the Indian
context in Hindi language. Additionally, we leverage LLMs including ChatGPT and
InstructGPT to augment our dataset with diverse societal biases and stereotypes
prevalent in India. The included bias dimensions encompass gender, religion,
caste, age, region, physical appearance, and occupation. We also build a
resource to address intersectional biases along three intersectional
dimensions. Our dataset contains 800 sentence pairs and 300 tuples for bias
measurement across different demographics. The dataset is available in English
and Hindi, providing a size comparable to existing benchmark datasets.
Furthermore, using IndiBias we compare ten different language models on
multiple bias measurement metrics. We observed that the language models exhibit
more bias across a majority of the intersectional groups.
