  Mitigating algorithmic bias is a critical task in the development and
deployment of machine learning models. While several toolkits exist to aid
machine learning practitioners in addressing fairness issues, little is known
about the strategies practitioners employ to evaluate model fairness and what
factors influence their assessment, particularly in the context of text
classification. Two common approaches of evaluating the fairness of a model are
group fairness and individual fairness. We run a study with Machine Learning
practitioners (n=24) to understand the strategies used to evaluate models.
Metrics presented to practitioners (group vs. individual fairness) impact which
models they consider fair. Participants focused on risks associated with
underpredicting/overpredicting and model sensitivity relative to identity token
manipulations. We discover fairness assessment strategies involving personal
experiences or how users form groups of identity tokens to test model fairness.
We provide recommendations for interactive tools for evaluating fairness in
text classification.
