  Navigating complex and dynamic environments requires autonomous vehicles
(AVs) to reason about both visible and occluded regions. This involves
predicting the future motion of observed agents, inferring occluded ones, and
modeling their interactions based on vectorized scene representations of the
partially observable environment. However, prior work on occlusion inference
and trajectory prediction have developed in isolation, with the former based on
simplified rasterized methods and the latter assuming full environment
observability. We introduce the Scene Informer, a unified approach for
predicting both observed agent trajectories and inferring occlusions in a
partially observable setting. It uses a transformer to aggregate various input
modalities and facilitate selective queries on occlusions that might intersect
with the AV's planned path. The framework estimates occupancy probabilities and
likely trajectories for occlusions, as well as forecast motion for observed
agents. We explore common observability assumptions in both domains and their
performance impact. Our approach outperforms existing methods in both occupancy
prediction and trajectory prediction in partially observable setting on the
Waymo Open Motion Dataset.
