  Although essential to revealing biased performance, well intentioned attempts
at algorithmic auditing can have effects that may harm the very populations
these measures are meant to protect. This concern is even more salient while
auditing biometric systems such as facial recognition, where the data is
sensitive and the technology is often used in ethically questionable manners.
We demonstrate a set of five ethical concerns in the particular case of
auditing commercial facial processing technology, highlighting additional
design considerations and ethical tensions the auditor needs to be aware of so
as not exacerbate or complement the harms propagated by the audited system. We
go further to provide tangible illustrations of these concerns, and conclude by
reflecting on what these concerns mean for the role of the algorithmic audit
and the fundamental product limitations they reveal.
