  AI based Face Recognition Systems (FRSs) are now widely distributed and
deployed as MLaaS solutions all over the world, moreso since the COVID-19
pandemic for tasks ranging from validating individuals' faces while buying SIM
cards to surveillance of citizens. Extensive biases have been reported against
marginalized groups in these systems and have led to highly discriminatory
outcomes. The post-pandemic world has normalized wearing face masks but FRSs
have not kept up with the changing times. As a result, these systems are
susceptible to mask based face occlusion. In this study, we audit four
commercial and nine open-source FRSs for the task of face re-identification
between different varieties of masked and unmasked images across five benchmark
datasets (total 14,722 images). These simulate a realistic
validation/surveillance task as deployed in all major countries around the
world. Three of the commercial and five of the open-source FRSs are highly
inaccurate; they further perpetuate biases against non-White individuals, with
the lowest accuracy being 0%. A survey for the same task with 85 human
participants also results in a low accuracy of 40%. Thus a human-in-the-loop
moderation in the pipeline does not alleviate the concerns, as has been
frequently hypothesized in literature. Our large-scale study shows that
developers, lawmakers and users of such services need to rethink the design
principles behind FRSs, especially for the task of face re-identification,
taking cognizance of observed biases.
