  Autonomous vehicle (AV) stacks are typically built in a modular fashion, with
explicit components performing detection, tracking, prediction, planning,
control, etc. While modularity improves reusability, interpretability, and
generalizability, it also suffers from compounding errors, information
bottlenecks, and integration challenges. To overcome these challenges, a
prominent approach is to convert the AV stack into an end-to-end neural network
and train it with data. While such approaches have achieved impressive results,
they typically lack interpretability and reusability, and they eschew
principled analytical components, such as planning and control, in favor of
deep neural networks. To enable the joint optimization of AV stacks while
retaining modularity, we present DiffStack, a differentiable and modular stack
for prediction, planning, and control. Crucially, our model-based planning and
control algorithms leverage recent advancements in differentiable optimization
to produce gradients, enabling optimization of upstream components, such as
prediction, via backpropagation through planning and control. Our results on
the nuScenes dataset indicate that end-to-end training with DiffStack yields
substantial improvements in open-loop and closed-loop planning metrics by,
e.g., learning to make fewer prediction errors that would affect planning.
Beyond these immediate benefits, DiffStack opens up new opportunities for fully
data-driven yet modular and interpretable AV architectures. Project website:
https://sites.google.com/view/diffstack
