  Online platforms face the challenge of moderating an ever-increasing volume
of content, including harmful hate speech. In the absence of clear legal
definitions and a lack of transparency regarding the role of algorithms in
shaping decisions on content moderation, there is a critical need for external
accountability. Our study contributes to filling this gap by systematically
evaluating four leading cloud-based content moderation services through a
third-party audit, highlighting issues such as biases against minorities and
vulnerable groups that may arise through over-reliance on these services. Using
a black-box audit approach and four benchmark data sets, we measure performance
in explicit and implicit hate speech detection as well as counterfactual
fairness through perturbation sensitivity analysis and present disparities in
performance for certain target identity groups and data sets. Our analysis
reveals that all services had difficulties detecting implicit hate speech,
which relies on more subtle and codified messages. Moreover, our results point
to the need to remove group-specific bias. It seems that biases towards some
groups, such as Women, have been mostly rectified, while biases towards other
groups, such as LGBTQ+ and PoC remain.
