  Novel view synthesis from a single image requires inferring occluded regions
of objects and scenes whilst simultaneously maintaining semantic and physical
consistency with the input. Existing approaches condition neural radiance
fields (NeRF) on local image features, projecting points to the input image
plane, and aggregating 2D features to perform volume rendering. However, under
severe occlusion, this projection fails to resolve uncertainty, resulting in
blurry renderings that lack details. In this work, we propose NerfDiff, which
addresses this issue by distilling the knowledge of a 3D-aware conditional
diffusion model (CDM) into NeRF through synthesizing and refining a set of
virtual views at test time. We further propose a novel NeRF-guided distillation
algorithm that simultaneously generates 3D consistent virtual views from the
CDM samples, and finetunes the NeRF based on the improved virtual views. Our
approach significantly outperforms existing NeRF-based and geometry-free
approaches on challenging datasets, including ShapeNet, ABO, and Clevr3D.
