  Recent studies show that Natural Language Processing (NLP) technologies
propagate societal biases about demographic groups associated with attributes
such as gender, race, and nationality. To create interventions and mitigate
these biases and associated harms, it is vital to be able to detect and measure
such biases. While existing works propose bias evaluation and mitigation
methods for various tasks, there remains a need to cohesively understand the
biases and the specific harms they measure, and how different measures compare
with each other. To address this gap, this work presents a practical framework
of harms and a series of questions that practitioners can answer to guide the
development of bias measures. As a validation of our framework and
documentation questions, we also present several case studies of how existing
bias measures in NLP -- both intrinsic measures of bias in representations and
extrinsic measures of bias of downstream applications -- can be aligned with
different harms and how our proposed documentation questions facilitates more
holistic understanding of what bias measures are measuring.
