  Machine learning techniques have been widely used in natural language
processing (NLP). However, as revealed by many recent studies, machine learning
models often inherit and amplify the societal biases in data. Various metrics
have been proposed to quantify biases in model predictions. In particular,
several of them evaluate disparity in model performance between protected
groups and advantaged groups in the test corpus. However, we argue that
evaluating bias at the corpus level is not enough for understanding how biases
are embedded in a model. In fact, a model with similar aggregated performance
between different groups on the entire data may behave differently on instances
in a local region. To analyze and detect such local bias, we propose LOGAN, a
new bias detection technique based on clustering. Experiments on toxicity
classification and object classification tasks show that LOGAN identifies bias
in a local region and allows us to better analyze the biases in model
predictions.
