  Scene labeling task is to segment the image into meaningful regions and
categorize them into classes of objects which comprised the image. Commonly
used methods typically find the local features for each segment and label them
using classifiers. Afterward, labeling is smoothed in order to make sure that
neighboring regions receive similar labels. However, they ignore expressive and
non-local dependencies among regions due to expensive training and inference.
In this paper, we propose to use high-level knowledge regarding rules in the
inference to incorporate dependencies among regions in the image to improve
scores of classification. Towards this aim, we extract these rules from data
and transform them into constraints for Integer Programming to optimize the
structured problem of assigning labels to super-pixels (consequently pixels) of
an image. In addition, we propose to use soft-constraints in some scenarios,
allowing violating the constraint by imposing a penalty, to make the model more
flexible. We assessed our approach on three datasets and obtained promising
results.
