  In the recent past, different researchers have proposed privacy-enhancing
face recognition systems designed to conceal soft-biometric attributes at
feature level. These works have reported impressive results, but generally did
not consider specific attacks in their analysis of privacy protection. We
introduce an attack on said schemes based on two observations: (1) highly
similar facial representations usually originate from face images with similar
soft-biometric attributes; (2) to achieve high recognition accuracy, robustness
against intra-class variations within facial representations has to be retained
in their privacy-enhanced versions. The presented attack only requires the
privacy-enhancing algorithm as a black-box and a relatively small database of
face images with annotated soft-biometric attributes. Firstly, an intercepted
privacy-enhanced face representation is compared against the attacker's
database. Subsequently, the unknown attribute is inferred from the attributes
associated with the highest obtained similarity scores. In the experiments, the
attack is applied against two state-of-the-art approaches. The attack is shown
to circumvent the privacy enhancement to a considerable degree and is able to
correctly classify gender with an accuracy of up to approximately 90%. Future
works on privacy-enhancing face recognition are encouraged to include the
proposed attack in evaluations on the privacy protection.
