  The knowledge replay technique has been widely used in many tasks such as
continual learning and continuous domain adaptation. The key lies in how to
effectively encode the knowledge extracted from previous data and replay them
during current training procedure. A simple yet effective model to achieve
knowledge replay is autoencoder. However, the number of stored latent codes in
autoencoder increases linearly with the scale of data and the trained encoder
is redundant for the replaying stage. In this paper, we propose a novel and
efficient knowledge recording network (KRNet) which directly maps an arbitrary
sample identity number to the corresponding datum. Compared with autoencoder,
our KRNet requires significantly ($400\times$) less storage cost for the latent
codes and can be trained without the encoder sub-network. Extensive experiments
validate the efficiency of KRNet, and as a showcase, it is successfully applied
in the task of continual learning.
