  In this work we present a novel approach to joint semantic localisation and
scene understanding. Our work is motivated by the need for localisation
algorithms which not only predict 6-DoF camera pose but also simultaneously
recognise surrounding objects and estimate 3D geometry. Such capabilities are
crucial for computer vision guided systems which interact with the environment:
autonomous driving, augmented reality and robotics. In particular, we propose a
two step procedure. During the first step we train a convolutional neural
network to jointly predict per-pixel globally unique instance labels and
corresponding local coordinates for each instance of a static object (e.g. a
building). During the second step we obtain scene coordinates by combining
object center coordinates and local coordinates and use them to perform 6-DoF
camera pose estimation. We evaluate our approach on real world (CamVid-360) and
artificial (SceneCity) autonomous driving datasets. We obtain smaller mean
distance and angular errors than state-of-the-art 6-DoF pose estimation
algorithms based on direct pose regression and pose estimation from scene
coordinates on all datasets. Our contributions include: (i) a novel formulation
of scene coordinate regression as two separate tasks of object instance
recognition and local coordinate regression and a demonstration that our
proposed solution allows to predict accurate 3D geometry of static objects and
estimate 6-DoF pose of camera on (ii) maps larger by several orders of
magnitude than previously attempted by scene coordinate regression methods, as
well as on (iii) lightweight, approximate 3D maps built from 3D primitives such
as building-aligned cuboids.
