  Facial expression is temporally dynamic event which can be decomposed into a
set of muscle motions occurring in different facial regions over various time
intervals. For dynamic expression recognition, two key issues, temporal
alignment and semantics-aware dynamic representation, must be taken into
account. In this paper, we attempt to solve both problems via manifold modeling
of videos based on a novel mid-level representation, i.e.
\textbf{expressionlet}. Specifically, our method contains three key stages: 1)
each expression video clip is characterized as a spatial-temporal manifold
(STM) formed by dense low-level features; 2) a Universal Manifold Model (UMM)
is learned over all low-level features and represented as a set of local modes
to statistically unify all the STMs. 3) the local modes on each STM can be
instantiated by fitting to UMM, and the corresponding expressionlet is
constructed by modeling the variations in each local mode. With above strategy,
expression videos are naturally aligned both spatially and temporally. To
enhance the discriminative power, the expressionlet-based STM representation is
further processed with discriminant embedding. Our method is evaluated on four
public expression databases, CK+, MMI, Oulu-CASIA, and FERA. In all cases, our
method outperforms the known state-of-the-art by a large margin.
