  Neural Implicit Representation (NIR) has recently gained significant
attention due to its remarkable ability to encode complex and high-dimensional
data into representation space and easily reconstruct it through a trainable
mapping function. However, NIR methods assume a one-to-one mapping between the
target data and representation models regardless of data relevancy or
similarity. This results in poor generalization over multiple complex data and
limits their efficiency and scalability. Motivated by continual learning, this
work investigates how to accumulate and transfer neural implicit
representations for multiple complex video data over sequential encoding
sessions. To overcome the limitation of NIR, we propose a novel method,
Progressive Fourier Neural Representation (PFNR), that aims to find an adaptive
and compact sub-module in Fourier space to encode videos in each training
session. This sparsified neural encoding allows the neural network to hold free
weights, enabling an improved adaptation for future videos. In addition, when
learning a representation for a new video, PFNR transfers the representation of
previous videos with frozen weights. This design allows the model to
continuously accumulate high-quality neural representations for multiple videos
while ensuring lossless decoding that perfectly preserves the learned
representations for previous videos. We validate our PFNR method on the UVG8/17
and DAVIS50 video sequence benchmarks and achieve impressive performance gains
over strong continual learning baselines. The PFNR code is available at
https://github.com/ihaeyong/PFNR.git.
