  Some of the most exciting experiences that Metaverse promises to offer, for
instance, live interactions with virtual characters in virtual environments,
require real-time photo-realistic rendering. 3D reconstruction approaches to
rendering, active or passive, still require extensive cleanup work to fix the
meshes or point clouds. In this paper, we present a neural volumography
technique called neural volumetric video or NeuVV to support immersive,
interactive, and spatial-temporal rendering of volumetric video contents with
photo-realism and in real-time. The core of NeuVV is to efficiently encode a
dynamic neural radiance field (NeRF) into renderable and editable primitives.
We introduce two types of factorization schemes: a hyper-spherical harmonics
(HH) decomposition for modeling smooth color variations over space and time and
a learnable basis representation for modeling abrupt density and color changes
caused by motion. NeuVV factorization can be integrated into a Video Octree
(VOctree) analogous to PlenOctree to significantly accelerate training while
reducing memory overhead. Real-time NeuVV rendering further enables a class of
immersive content editing tools. Specifically, NeuVV treats each VOctree as a
primitive and implements volume-based depth ordering and alpha blending to
realize spatial-temporal compositions for content re-purposing. For example, we
demonstrate positioning varied manifestations of the same performance at
different 3D locations with different timing, adjusting color/texture of the
performer's clothing, casting spotlight shadows and synthesizing distance
falloff lighting, etc, all at an interactive speed. We further develop a hybrid
neural-rasterization rendering framework to support consumer-level VR headsets
so that the aforementioned volumetric video viewing and editing, for the first
time, can be conducted immersively in virtual 3D space.
