  Structure from motion (SfM) enables us to reconstruct a scene via casual
capture from cameras at different viewpoints, and novel view synthesis (NVS)
allows us to render a captured scene from a new viewpoint. Both are hard with
casual capture and dynamic scenes: SfM produces noisy and spatio-temporally
sparse reconstructed point clouds, resulting in NVS with spatio-temporally
inconsistent effects. We consider SfM and NVS parts together to ease the
challenge. First, for SfM, we recover stable camera poses, then we defer the
requirement for temporally-consistent points across the scene and reconstruct
only a sparse point cloud per timestep that is noisy in space-time. Second, for
NVS, we present a variational diffusion formulation on depths and colors that
lets us robustly cope with the noise by enforcing spatio-temporal consistency
via per-pixel reprojection weights derived from the input views. Together, this
deferred approach generates novel views for dynamic scenes without requiring
challenging spatio-temporally consistent reconstructions nor training complex
models on large datasets. We demonstrate our algorithm on real-world dynamic
scenes against classic and more recent learning-based baseline approaches.
