  The popularity of Neural Radiance Fields (NeRFs) for view synthesis has led
to a desire for NeRF editing tools. Here, we focus on inpainting regions in a
view-consistent and controllable manner. In addition to the typical NeRF inputs
and masks delineating the unwanted region in each view, we require only a
single inpainted view of the scene, i.e., a reference view. We use monocular
depth estimators to back-project the inpainted view to the correct 3D
positions. Then, via a novel rendering technique, a bilateral solver can
construct view-dependent effects in non-reference views, making the inpainted
region appear consistent from any view. For non-reference disoccluded regions,
which cannot be supervised by the single reference view, we devise a method
based on image inpainters to guide both the geometry and appearance. Our
approach shows superior performance to NeRF inpainting baselines, with the
additional advantage that a user can control the generated scene via a single
inpainted image. Project page: https://ashmrz.github.io/reference-guided-3d
