  As the expressive depth of an emotional face differs with individuals or
expressions, recognizing an expression using a single facial image at a moment
is difficult. A relative expression of a query face compared to a reference
face might alleviate this difficulty. In this paper, we propose to utilize
contrastive representation that embeds a distinctive expressive factor for a
discriminative purpose. The contrastive representation is calculated at the
embedding layer of deep networks by comparing a given (query) image with the
reference image. We attempt to utilize a generative reference image that is
estimated based on the given image. Consequently, we deploy deep neural
networks that embed a combination of a generative model, a contrastive model,
and a discriminative model with an end-to-end training manner. In our proposed
networks, we attempt to disentangle a facial expressive factor in two steps
including learning of a generator network and a contrastive encoder network. We
conducted extensive experiments on publicly available face expression databases
(CK+, MMI, Oulu-CASIA, and in-the-wild databases) that have been widely adopted
in the recent literatures. The proposed method outperforms the known
state-of-the art methods in terms of the recognition accuracy.
