  In this paper, we propose the Deep Structured self-Driving Network (DSDNet),
which performs object detection, motion prediction, and motion planning with a
single neural network. Towards this goal, we develop a deep structured energy
based model which considers the interactions between actors and produces
socially consistent multimodal future predictions. Furthermore, DSDNet
explicitly exploits the predicted future distributions of actors to plan a safe
maneuver by using a structured planning cost. Our sample-based formulation
allows us to overcome the difficulty in probabilistic inference of continuous
random variables. Experiments on a number of large-scale self driving datasets
demonstrate that our model significantly outperforms the state-of-the-art.
