  This paper deals with the utterance-level modalities missing problem with
uncertain patterns on emotion recognition in conversation (ERC) task. Present
models generally predict the speaker's emotions by its current utterance and
context, which is degraded by modality missing considerably. Our work proposes
a framework Missing-Modality Robust emotion Recognition (M2R2), which trains
emotion recognition model with iterative data augmentation by learned common
representation. Firstly, a network called Party Attentive Network (PANet) is
designed to classify emotions, which tracks all the speakers' states and
context. Attention mechanism between speaker with other participants and
dialogue topic is used to decentralize dependence on multi-time and multi-party
utterances instead of the possible incomplete one. Moreover, the Common
Representation Learning (CRL) problem is defined for modality-missing problem.
Data imputation methods improved by the adversarial strategy are used here to
construct extra features to augment data. Extensive experiments and case
studies validate the effectiveness of our methods over baselines for
modality-missing emotion recognition on two different datasets.
