  This paper targets to explore the inter-subject variations eliminated facial
expression representation in the compressed video domain. Most of the previous
methods process the RGB images of a sequence, while the off-the-shelf and
valuable expression-related muscle movement already embedded in the compression
format. In the up to two orders of magnitude compressed domain, we can
explicitly infer the expression from the residual frames and possible to
extract identity factors from the I frame with a pre-trained face recognition
network. By enforcing the marginal independent of them, the expression feature
is expected to be purer for the expression and be robust to identity shifts. We
do not need the identity label or multiple expression samples from the same
person for identity elimination. Moreover, when the apex frame is annotated in
the dataset, the complementary constraint can be further added to regularize
the feature-level game. In testing, only the compressed residual frames are
required to achieve expression prediction. Our solution can achieve comparable
or better performance than the recent decoded image based methods on the
typical FER benchmarks with about 3$\times$ faster inference with compressed
data.
