  In recent years, deep learning has greatly streamlined the process of
generating realistic fake face images. Aware of the dangers, researchers have
developed various tools to spot these counterfeits. Yet none asked the
fundamental question: What digital manipulations make a real photographic face
image fake, while others do not? In this paper, we put face forgery in a
semantic context and define that computational methods that alter semantic face
attributes to exceed human discrimination thresholds are sources of face
forgery. Guided by our new definition, we construct a large face forgery image
dataset, where each image is associated with a set of labels organized in a
hierarchical graph. Our dataset enables two new testing protocols to probe the
generalization of face forgery detectors. Moreover, we propose a
semantics-oriented face forgery detection method that captures label relations
and prioritizes the primary task (\ie, real or fake face detection). We show
that the proposed dataset successfully exposes the weaknesses of current
detectors as the test set and consistently improves their generalizability as
the training set. Additionally, we demonstrate the superiority of our
semantics-oriented method over traditional binary and multi-class
classification-based detectors.
