  Neural rendering is fuelling a unification of learning, 3D geometry and video
understanding that has been waiting for more than two decades. Progress,
however, is still hampered by a lack of suitable datasets and benchmarks. To
address this gap, we introduce EPIC Fields, an augmentation of EPIC-KITCHENS
with 3D camera information. Like other datasets for neural rendering, EPIC
Fields removes the complex and expensive step of reconstructing cameras using
photogrammetry, and allows researchers to focus on modelling problems. We
illustrate the challenge of photogrammetry in egocentric videos of dynamic
actions and propose innovations to address them. Compared to other neural
rendering datasets, EPIC Fields is better tailored to video understanding
because it is paired with labelled action segments and the recent VISOR segment
annotations. To further motivate the community, we also evaluate two benchmark
tasks in neural rendering and segmenting dynamic objects, with strong baselines
that showcase what is not possible today. We also highlight the advantage of
geometry in semi-supervised video object segmentations on the VISOR
annotations. EPIC Fields reconstructs 96% of videos in EPICKITCHENS,
registering 19M frames in 99 hours recorded in 45 kitchens.
