  Can deep learning models achieve greater generalization if their training is
guided by reference to human perceptual abilities? And how can we implement
this in a practical manner? This paper proposes a training strategy to ConveY
Brain Oversight to Raise Generalization (CYBORG). This new approach
incorporates human-annotated saliency maps into a loss function that guides the
model's learning to focus on image regions that humans deem salient for the
task. The Class Activation Mapping (CAM) mechanism is used to probe the model's
current saliency in each training batch, juxtapose this model saliency with
human saliency, and penalize large differences. Results on the task of
synthetic face detection, selected to illustrate the effectiveness of the
approach, show that CYBORG leads to significant improvement in accuracy on
unseen samples consisting of face images generated from six Generative
Adversarial Networks across multiple classification network architectures. We
also show that scaling to even seven times the training data, or using
non-human-saliency auxiliary information, such as segmentation masks, and
standard loss cannot beat the performance of CYBORG-trained models. As a side
effect of this work, we observe that the addition of explicit region annotation
to the task of synthetic face detection increased human classification
accuracy. This work opens a new area of research on how to incorporate human
visual saliency into loss functions in practice. All data, code and pre-trained
models used in this work are offered with this paper.
