  This report examines the Pinned AUC metric introduced and highlights some of
its limitations. Pinned AUC provides a threshold-agnostic measure of unintended
bias in a classification model, inspired by the ROC-AUC metric. However, as we
highlight in this report, there are ways that the metric can obscure different
kinds of unintended biases when the underlying class distributions on which
bias is being measured are not carefully controlled.
