  An automated vehicle operating in an urban environment must be able to
perceive and recognise object/obstacles in a three-dimensional world while
navigating in a constantly changing environment. In order to plan and execute
accurate sophisticated driving maneuvers, a high-level contextual understanding
of the surroundings is essential. Due to the recent progress in image
processing, it is now possible to obtain high definition semantic information
in 2D from monocular cameras, though cameras cannot reliably provide the highly
accurate 3D information provided by lasers. The fusion of these two sensor
modalities can overcome the shortcomings of each individual sensor, though
there are a number of important challenges that need to be addressed in a
probabilistic manner. In this paper, we address the common, yet challenging,
lidar/camera/semantic fusion problems which are seldom approached in a wholly
probabilistic manner. Our approach is capable of using a multi-sensor platform
to build a three-dimensional semantic voxelized map that considers the
uncertainty of all of the processes involved. We present a probabilistic
pipeline that incorporates uncertainties from the sensor readings (cameras,
lidar, IMU and wheel encoders), compensation for the motion of the vehicle, and
heuristic label probabilities for the semantic images. We also present a novel
and efficient viewpoint validation algorithm to check for occlusions from the
camera frames. A probabilistic projection is performed from the camera images
to the lidar point cloud. Each labelled lidar scan then feeds into an octree
map building algorithm that updates the class probabilities of the map voxels
every time a new observation is available. We validate our approach using a set
of qualitative and quantitative experimental tests on the USyd Dataset.
