  In order for NLP technology to be widely applicable, fair, and useful, it
needs to serve a diverse set of speakers across the world's languages, be
equitable, i.e., not unduly biased towards any particular language, and be
inclusive of all users, particularly in low-resource settings where compute
constraints are common. In this paper, we propose an evaluation paradigm that
assesses NLP technologies across all three dimensions. While diversity and
inclusion have received attention in recent literature, equity is currently
unexplored. We propose to address this gap using the Gini coefficient, a
well-established metric used for estimating societal wealth inequality. Using
our paradigm, we highlight the distressed state of current technologies for
Indian (IN) languages (a linguistically large and diverse set, with a varied
speaker population), across all three dimensions. To improve upon these
metrics, we demonstrate the importance of region-specific choices in model
building and dataset creation, and more importantly, propose a novel,
generalisable approach to optimal resource allocation during fine-tuning.
Finally, we discuss steps to mitigate these biases and encourage the community
to employ multi-faceted evaluation when building linguistically diverse and
equitable technologies.
