  Large language models (LLMs) can exhibit bias in a variety of ways. Such
biases can create or exacerbate unfair outcomes for certain groups within a
protected attribute, including, but not limited to sex, race, sexual
orientation, or age. This paper aims to provide a technical guide for
practitioners to assess bias and fairness risks in LLM use cases. The main
contribution of this work is a decision framework that allows practitioners to
determine which metrics to use for a specific LLM use case. To achieve this,
this study categorizes LLM bias and fairness risks, maps those risks to a
taxonomy of LLM use cases, and then formally defines various metrics to assess
each type of risk. As part of this work, several new bias and fairness metrics
are introduced, including innovative counterfactual metrics as well as metrics
based on stereotype classifiers. Instead of focusing solely on the model
itself, the sensitivity of both prompt-risk and model-risk are taken into
account by defining evaluations at the level of an LLM use case, characterized
by a model and a population of prompts. Furthermore, because all of the
evaluation metrics are calculated solely using the LLM output, the proposed
framework is highly practical and easily actionable for practitioners.
