  We propose CARFF, a method for predicting future 3D scenes given past
observations. Our method maps 2D ego-centric images to a distribution over
plausible 3D latent scene configurations and predicts the evolution of
hypothesized scenes through time. Our latents condition a global Neural
Radiance Field (NeRF) to represent a 3D scene model, enabling explainable
predictions and straightforward downstream planning. This approach models the
world as a POMDP and considers complex scenarios of uncertainty in
environmental states and dynamics. Specifically, we employ a two-stage training
of Pose-Conditional-VAE and NeRF to learn 3D representations, and
auto-regressively predict latent scene representations utilizing a mixture
density network. We demonstrate the utility of our method in scenarios using
the CARLA driving simulator, where CARFF enables efficient trajectory and
contingency planning in complex multi-agent autonomous driving scenarios
involving occlusions.
