  Localization is an essential task for mobile autonomous robotic systems that
want to use pre-existing maps or create new ones in the context of SLAM. Today,
many robotic platforms are equipped with high-accuracy 3D LiDAR sensors, which
allow a geometric mapping, and cameras able to provide semantic cues of the
environment. Segment-based mapping and localization have been applied with
great success to 3D point-cloud data, while semantic understanding has been
shown to improve localization performance in vision based systems. In this
paper we combine both modalities in SemSegMap, extending SegMap into a segment
based mapping framework able to also leverage color and semantic data from the
environment to improve localization accuracy and robustness. In particular, we
present new segmentation and descriptor extraction processes. The segmentation
process benefits from additional distance information from color and semantic
class consistency resulting in more repeatable segments and more overlap after
re-visiting a place. For the descriptor, a tight fusion approach in a
deep-learned descriptor extraction network is performed leading to a higher
descriptiveness for landmark matching. We demonstrate the advantages of this
fusion on multiple simulated and real-world datasets and compare its
performance to various baselines. We show that we are able to find 50.9% more
high-accuracy prior-less global localizations compared to SegMap on challenging
datasets using very compact maps while also providing accurate full 6 DoF pose
estimates in real-time.
