  Behavior prediction in dynamic, multi-agent systems is an important problem
in the context of self-driving cars, due to the complex representations and
interactions of road components, including moving agents (e.g. pedestrians and
vehicles) and road context information (e.g. lanes, traffic lights). This paper
introduces VectorNet, a hierarchical graph neural network that first exploits
the spatial locality of individual road components represented by vectors and
then models the high-order interactions among all components. In contrast to
most recent approaches, which render trajectories of moving agents and road
context information as bird-eye images and encode them with convolutional
neural networks (ConvNets), our approach operates on a vector representation.
By operating on the vectorized high definition (HD) maps and agent
trajectories, we avoid lossy rendering and computationally intensive ConvNet
encoding steps. To further boost VectorNet's capability in learning context
features, we propose a novel auxiliary task to recover the randomly masked out
map entities and agent trajectories based on their context. We evaluate
VectorNet on our in-house behavior prediction benchmark and the recently
released Argoverse forecasting dataset. Our method achieves on par or better
performance than the competitive rendering approach on both benchmarks while
saving over 70% of the model parameters with an order of magnitude reduction in
FLOPs. It also outperforms the state of the art on the Argoverse dataset.
