  Neural radiance fields (NeRFs) have gained popularity across various
applications. However, they face challenges in the sparse view setting, lacking
sufficient constraints from volume rendering. Reconstructing and understanding
a 3D scene from sparse and unconstrained cameras is a long-standing problem in
classical computer vision with diverse applications. While recent works have
explored NeRFs in sparse, unconstrained view scenarios, their focus has been
primarily on enhancing reconstruction and novel view synthesis. Our approach
takes a broader perspective by posing the question: "from where has each point
been seen?" -- which gates how well we can understand and reconstruct it. In
other words, we aim to determine the origin or provenance of each 3D point and
its associated information under sparse, unconstrained views. We introduce
ProvNeRF, a model that enriches a traditional NeRF representation by
incorporating per-point provenance, modeling likely source locations for each
point. We achieve this by extending implicit maximum likelihood estimation
(IMLE) for stochastic processes. Notably, our method is compatible with any
pre-trained NeRF model and the associated training camera poses. We demonstrate
that modeling per-point provenance offers several advantages, including
uncertainty estimation, criteria-based view selection, and improved novel view
synthesis, compared to state-of-the-art methods. Please visit our project page
at https://provnerf.github.io
