  For autonomous robots to navigate a complex environment, it is crucial to
understand the surrounding scene both geometrically and semantically. Modern
autonomous robots employ multiple sets of sensors, including lidars, radars,
and cameras. Managing the different reference frames and characteristics of the
sensors, and merging their observations into a single representation
complicates perception. Choosing a single unified representation for all
sensors simplifies the task of perception and fusion. In this work, we present
an end-to-end pipeline that performs semantic segmentation and short term
prediction using a top-down representation. Our approach consists of an
ensemble of neural networks which take in sensor data from different sensor
modalities and transform them into a single common top-down semantic grid
representation. We find this representation favorable as it is agnostic to
sensor-specific reference frames and captures both the semantic and geometric
information for the surrounding scene. Because the modalities share a single
output representation, they can be easily aggregated to produce a fused output.
In this work we predict short-term semantic grids but the framework can be
extended to other tasks. This approach offers a simple, extensible, end-to-end
approach for multi-modal perception and prediction.
