  Emotion recognition is a core research area at the intersection of artificial
intelligence and human communication analysis. It is a significant technical
challenge since humans display their emotions through complex idiosyncratic
combinations of the language, visual and acoustic modalities. In contrast to
traditional multimodal fusion techniques, we approach emotion recognition from
both direct person-independent and relative person-dependent perspectives. The
direct person-independent perspective follows the conventional emotion
recognition approach which directly infers absolute emotion labels from
observed multimodal features. The relative person-dependent perspective
approaches emotion recognition in a relative manner by comparing partial video
segments to determine if there was an increase or decrease in emotional
intensity. Our proposed model integrates these direct and relative prediction
perspectives by dividing the emotion recognition task into three easier
subtasks. The first subtask involves a multimodal local ranking of relative
emotion intensities between two short segments of a video. The second subtask
uses local rankings to infer global relative emotion ranks with a Bayesian
ranking algorithm. The third subtask incorporates both direct predictions from
observed multimodal behaviors and relative emotion ranks from local-global
rankings for final emotion prediction. Our approach displays excellent
performance on an audio-visual emotion recognition benchmark and improves over
other algorithms for multimodal fusion.
