  We propose a novel approach for modeling semantic contextual relationships in
videos. This graph-based model enables the learning and propagation of
higher-level spatial-temporal contexts to facilitate the semantic labeling of
local regions. We introduce an exemplar-based nonparametric view of contextual
cues, where the inherent relationships implied by object hypotheses are encoded
on a similarity graph of regions. Contextual relationships learning and
propagation are performed to estimate the pairwise contexts between all pairs
of unlabeled local regions. Our algorithm integrates the learned contexts into
a Conditional Random Field (CRF) in the form of pairwise potentials and infers
the per-region semantic labels. We evaluate our approach on the challenging
YouTube-Objects dataset which shows that the proposed contextual relationship
model outperforms the state-of-the-art methods.
