  State-of-the-art deep CNN face matchers are typically created using extensive
training sets of color face images. Our study reveals that such matchers attain
virtually identical accuracy when trained on either grayscale or color versions
of the training set, even when the evaluation is done using color test images.
Furthermore, we demonstrate that shallower models, lacking the capacity to
model complex representations, rely more heavily on low-level features such as
those associated with color. As a result, they display diminished accuracy when
trained with grayscale images. We then consider possible causes for deeper CNN
face matchers "not seeing color". Popular web-scraped face datasets actually
have 30 to 60% of their identities with one or more grayscale images. We
analyze whether this grayscale element in the training set impacts the accuracy
achieved, and conclude that it does not. We demonstrate that using only
grayscale images for both training and testing achieves accuracy comparable to
that achieved using only color images for deeper models. This holds true for
both real and synthetic training datasets. HSV color space, which separates
chroma and luma information, does not improve the network's learning about
color any more than in the RGB color space. We then show that the skin region
of an individual's images in a web-scraped training set exhibits significant
variation in their mapping to color space. This suggests that color carries
limited identity-specific information. We also show that when the first
convolution layer is restricted to a single filter, models learn a grayscale
conversion filter and pass a grayscale version of the input color image to the
next layer. Finally, we demonstrate that leveraging the lower per-image storage
for grayscale to increase the number of images in the training set can improve
accuracy of the face recognition model.
