  Neural radiance field (NeRF) has achieved great success in novel view
synthesis and 3D representation for static scenarios. Existing dynamic NeRFs
usually exploit a locally dense grid to fit the deformation field; however,
they fail to capture the global dynamics and concomitantly yield models of
heavy parameters. We observe that the 4D space is inherently sparse. Firstly,
the deformation field is sparse in spatial but dense in temporal due to the
continuity of of motion. Secondly, the radiance field is only valid on the
surface of the underlying scene, usually occupying a small fraction of the
whole space. We thus propose to represent the 4D scene using a learnable sparse
latent space, a.k.a. SLS4D. Specifically, SLS4D first uses dense learnable time
slot features to depict the temporal space, from which the deformation field is
fitted with linear multi-layer perceptions (MLP) to predict the displacement of
a 3D position at any time. It then learns the spatial features of a 3D position
using another sparse latent space. This is achieved by learning the adaptive
weights of each latent code with the attention mechanism. Extensive experiments
demonstrate the effectiveness of our SLS4D: it achieves the best 4D novel view
synthesis using only about $6\%$ parameters of the most recent work.
