  The field of automated machine learning (AutoML) introduces techniques that
automate parts of the development of machine learning (ML) systems,
accelerating the process and reducing barriers for novices. However, decisions
derived from ML models can reproduce, amplify, or even introduce unfairness in
our societies, causing harm to (groups of) individuals. In response,
researchers have started to propose AutoML systems that jointly optimize
fairness and predictive performance to mitigate fairness-related harm. However,
fairness is a complex and inherently interdisciplinary subject, and solely
posing it as an optimization problem can have adverse side effects. With this
work, we aim to raise awareness among developers of AutoML systems about such
limitations of fairness-aware AutoML, while also calling attention to the
potential of AutoML as a tool for fairness research. We present a comprehensive
overview of different ways in which fairness-related harm can arise and the
ensuing implications for the design of fairness-aware AutoML. We conclude that
while fairness cannot be automated, fairness-aware AutoML can play an important
role in the toolbox of ML practitioners. We highlight several open technical
challenges for future work in this direction. Additionally, we advocate for the
creation of more user-centered assistive systems designed to tackle challenges
encountered in fairness work
