  This paper presents a method for learning overcomplete dictionaries composed
of two modalities that describe a 3D scene: image intensity and scene depth. We
propose a novel Joint Basis Pursuit (JBP) algorithm that finds related sparse
features in two modalities using conic programming and integrate it into a
two-step dictionary learning algorithm. JBP differs from related convex
algorithms because it finds joint sparsity models with different atoms and
different coefficient values for intensity and depth. This is crucial for
recovering generative models where the same sparse underlying causes (3D
features) give rise to different signals (intensity and depth). We give a
theoretical bound for the sparse coefficient recovery error obtained by JBP,
and show experimentally that JBP is far superior to the state of the art Group
Lasso algorithm. When applied to the Middlebury depth-intensity database, our
learning algorithm converges to a set of related features, such as pairs of
depth and intensity edges or image textures and depth slants. Finally, we show
that the learned dictionary and JBP achieve the state of the art depth
inpainting performance on time-of-flight 3D data.
