  The impact of AI models on marginalized communities has traditionally been
measured by identifying performance differences between specified demographic
subgroups. Though this approach aims to center vulnerable groups, it risks
obscuring patterns of harm faced by intersectional subgroups or shared across
multiple groups. To address this, we draw on theories of marginalization from
disability studies and related disciplines, which state that people farther
from the norm face greater adversity, to consider the "margins" in the domain
of toxicity detection. We operationalize the "margins" of a dataset by
employing outlier detection to identify text about people with demographic
attributes distant from the "norm". We find that model performance is
consistently worse for demographic outliers, with mean squared error (MSE)
between outliers and non-outliers up to 70.4% worse across toxicity types. It
is also worse for text outliers, with a MSE up to 68.4% higher for outliers
than non-outliers. We also find text and demographic outliers to be
particularly susceptible to errors in the classification of severe toxicity and
identity attacks. Compared to analysis of disparities using traditional
demographic breakdowns, we find that our outlier analysis frequently surfaces
greater harms faced by a larger, more intersectional group, which suggests that
outlier analysis is particularly beneficial for identifying harms against those
groups.
