  Detecting anomalous faces has important applications. For example, a system
might tell when a train driver is incapacitated by a medical event, and assist
in adopting a safe recovery strategy. These applications are demanding, because
they require accurate detection of rare anomalies that may be seen only at
runtime. Such a setting causes supervised methods to perform poorly. We
describe a method for detecting an anomalous face image that meets these
requirements. We construct a feature vector that reliably has large entries for
anomalous images, then use various simple unsupervised methods to score the
image based on the feature. Obvious constructions (autoencoder codes;
autoencoder residuals) are defeated by a 'peeking' behavior in autoencoders.
Our feature construction removes rectangular patches from the image, predicts
the likely content of the patch conditioned on the rest of the image using a
specially trained autoencoder, then compares the result to the image. High
scores suggest that the patch was difficult for an autoencoder to predict, and
so is likely anomalous. We demonstrate that our method can identify real
anomalous face images in pools of typical images, taken from celeb-A, that is
much larger than usual in state-of-the-art experiments. A control experiment
based on our method with another set of normal celebrity images - a 'typical
set', but nonceleb-A are not identified as anomalous; confirms this is not due
to special properties of celeb-A.
