  Detecting multiple planes in images is a challenging problem, but one with
many applications. Recent work such as J-Linkage and Ordered Residual Kernels
have focussed on developing a domain independent approach to detect multiple
structures. These multiple structure detection methods are then used for
estimating multiple homographies given feature matches between two images.
Features participating in the multiple homographies detected, provide us the
multiple scene planes. We show that these methods provide locally optimal
results and fail to merge detected planar patches to the true scene planes.
These methods use only residues obtained on applying homography of one plane to
another as cue for merging. In this paper, we develop additional cues such as
local consistency of planes, local normals, texture etc. to perform better
classification and merging . We formulate the classification as an MRF problem
and use TRWS message passing algorithm to solve non metric energy terms and
complex sparse graph structure. We show results on challenging dataset common
in robotics navigation scenarios where our method shows accuracy of more than
85 percent on average while being close or same as the actual number of scene
planes.
