  How to extract effective expression representations that invariant to the
identity-specific attributes is a long-lasting problem for facial expression
recognition (FER). Most of the previous methods process the RGB images of a
sequence, while we argue that the off-the-shelf and valuable expression-related
muscle movement is already embedded in the compression format. In this paper,
we target to explore the inter-subject variations eliminated facial expression
representation in the compressed video domain. In the up to two orders of
magnitude compressed domain, we can explicitly infer the expression from the
residual frames and possibly extract identity factors from the I frame with a
pre-trained face recognition network. By enforcing the marginal independence of
them, the expression feature is expected to be purer for the expression and be
robust to identity shifts. Specifically, we propose a novel collaborative
min-min game for mutual information (MI) minimization in latent space. We do
not need the identity label or multiple expression samples from the same person
for identity elimination. Moreover, when the apex frame is annotated in the
dataset, the complementary constraint can be further added to regularize the
feature-level game. In testing, only the compressed residual frames are
required to achieve expression prediction. Our solution can achieve comparable
or better performance than the recent decoded image-based methods on the
typical FER benchmarks with about 3 times faster inference.
