  Given a monocular video, segmenting and decoupling dynamic objects while
recovering the static environment is a widely studied problem in machine
intelligence. Existing solutions usually approach this problem in the image
domain, limiting their performance and understanding of the environment. We
introduce Decoupled Dynamic Neural Radiance Field (D$^2$NeRF), a
self-supervised approach that takes a monocular video and learns a 3D scene
representation which decouples moving objects, including their shadows, from
the static background. Our method represents the moving objects and the static
background by two separate neural radiance fields with only one allowing for
temporal changes. A naive implementation of this approach leads to the dynamic
component taking over the static one as the representation of the former is
inherently more general and prone to overfitting. To this end, we propose a
novel loss to promote correct separation of phenomena. We further propose a
shadow field network to detect and decouple dynamically moving shadows. We
introduce a new dataset containing various dynamic objects and shadows and
demonstrate that our method can achieve better performance than
state-of-the-art approaches in decoupling dynamic and static 3D objects,
occlusion and shadow removal, and image segmentation for moving objects.
