  A fundamental tenet of pattern recognition is that overlap between training
and testing sets causes an optimistic accuracy estimate. Deep CNNs for face
recognition are trained for N-way classification of the identities in the
training set. Accuracy is commonly estimated as average 10-fold classification
accuracy on image pairs from test sets such as LFW, CALFW, CPLFW, CFP-FP and
AgeDB-30. Because train and test sets have been independently assembled, images
and identities in any given test set may also be present in any given training
set. In particular, our experiments reveal a surprising degree of identity and
image overlap between the LFW family of test sets and the MS1MV2 training set.
Our experiments also reveal identity label noise in MS1MV2. We compare accuracy
achieved with same-size MS1MV2 subsets that are identity-disjoint and not
identity-disjoint with LFW, to reveal the size of the optimistic bias. Using
more challenging test sets from the LFW family, we find that the size of the
optimistic bias is larger for more challenging test sets. Our results highlight
the lack of and the need for identity-disjoint train and test methodology in
face recognition research.
