  We present a system for applying sim2real approaches to "in the wild" scenes
with realistic visuals, and to policies which rely on active perception using
RGB cameras. Given a short video of a static scene collected using a generic
phone, we learn the scene's contact geometry and a function for novel view
synthesis using a Neural Radiance Field (NeRF). We augment the NeRF rendering
of the static scene by overlaying the rendering of other dynamic objects (e.g.
the robot's own body, a ball). A simulation is then created using the rendering
engine in a physics simulator which computes contact dynamics from the static
scene geometry (estimated from the NeRF volume density) and the dynamic
objects' geometry and physical properties (assumed known). We demonstrate that
we can use this simulation to learn vision-based whole body navigation and ball
pushing policies for a 20 degrees of freedom humanoid robot with an actuated
head-mounted RGB camera, and we successfully transfer these policies to a real
robot. Project video is available at
https://sites.google.com/view/nerf2real/home
