  Identity-invariant facial expression recognition (FER) has been one of the
challenging computer vision tasks. Since conventional FER schemes do not
explicitly address the inter-identity variation of facial expressions, their
neural network models still operate depending on facial identity. This paper
proposes to quantify the inter-identity variation by utilizing pairs of similar
expressions explored through a specific matching process. We formulate the
identity matching process as an Optimal Transport (OT) problem. Specifically,
to find pairs of similar expressions from different identities, we define the
inter-feature similarity as a transportation cost. Then, optimal identity
matching to find the optimal flow with minimum transportation cost is performed
by Sinkhorn-Knopp iteration. The proposed matching method is not only easy to
plug in to other models, but also requires only acceptable computational
overhead. Extensive simulations prove that the proposed FER method improves the
PCC/CCC performance by up to 10\% or more compared to the runner-up on wild
datasets. The source code and software demo are available at
https://github.com/kdhht2334/ELIM_FER.
