  We adopt Convolutional Neural Networks (CNNs) to be our parametric model to
learn discriminative features and classifiers for local patch classification.
Based on the occurrence frequency distribution of classes, an ensemble of CNNs
(CNN-Ensemble) are learned, in which each CNN component focuses on learning
different and complementary visual patterns. The local beliefs of pixels are
output by CNN-Ensemble. Considering that visually similar pixels are
indistinguishable under local context, we leverage the global scene semantics
to alleviate the local ambiguity. The global scene constraint is mathematically
achieved by adding a global energy term to the labeling energy function, and it
is practically estimated in a non-parametric framework. A large margin based
CNN metric learning method is also proposed for better global belief
estimation. In the end, the integration of local and global beliefs gives rise
to the class likelihood of pixels, based on which maximum marginal inference is
performed to generate the label prediction maps. Even without any
post-processing, we achieve state-of-the-art results on the challenging
SiftFlow and Barcelona benchmarks.
