  Combinatorial optimization is frequently used in computer vision. For
instance, in applications like semantic segmentation, human pose estimation and
action recognition, programs are formulated for solving inference in
Conditional Random Fields (CRFs) to produce a structured output that is
consistent with visual features of the image. However, solving inference in
CRFs is in general intractable, and approximation methods are computationally
demanding and limited to unary, pairwise and hand-crafted forms of higher order
potentials. In this paper, we show that we can learn program heuristics, i.e.,
policies, for solving inference in higher order CRFs for the task of semantic
segmentation, using reinforcement learning. Our method solves inference tasks
efficiently without imposing any constraints on the form of the potentials. We
show compelling results on the Pascal VOC and MOTS datasets.
