  Language representations are efficient tools used across NLP applications,
but they are strife with encoded societal biases. These biases are studied
extensively, but with a primary focus on English language representations and
biases common in the context of Western society. In this work, we investigate
biases present in Hindi language representations with focuses on caste and
religion-associated biases. We demonstrate how biases are unique to specific
language representations based on the history and culture of the region they
are widely spoken in, and how the same societal bias (such as binary
gender-associated biases) is encoded by different words and text spans across
languages. The discoveries of our work highlight the necessity of culture
awareness and linguistic artifacts when modeling language representations, in
order to better understand the encoded biases.
