  For intelligent robotics applications, extending 3D mapping to 3D semantic
mapping enables robots to, not only localize themselves with respect to the
scene's geometrical features but also simultaneously understand the higher
level meaning of the scene contexts. Most previous methods focus on geometric
3D reconstruction and scene understanding independently notwithstanding the
fact that joint estimation can boost the accuracy of the semantic mapping. In
this paper, a dense RGB-D semantic mapping system with a Pixel-Voxel network is
proposed, which can perform dense 3D mapping while simultaneously recognizing
and semantically labelling each point in the 3D map. The proposed Pixel-Voxel
network obtains global context information by using PixelNet to exploit the RGB
image and meanwhile, preserves accurate local shape information by using
VoxelNet to exploit the corresponding 3D point cloud. Unlike the existing
architecture that fuses score maps from different models with equal weights, we
proposed a Softmax weighted fusion stack that adaptively learns the varying
contributions of PixelNet and VoxelNet, and fuses the score maps of the two
models according to their respective confidence levels. The proposed
Pixel-Voxel network achieves the state-of-the-art semantic segmentation
performance on the SUN RGB-D benchmark dataset. The runtime of the proposed
system can be boosted to 11-12Hz, enabling near to real-time performance using
an i7 8-cores PC with Titan X GPU.
