  New knowledge originates from the old. The various types of elements,
deposited in the training history, are a large amount of wealth for improving
learning deep models. In this survey, we comprehensively review and summarize
the topic--``Historical Learning: Learning Models with Learning History'',
which learns better neural models with the help of their learning history
during its optimization, from three detailed aspects: Historical Type (what),
Functional Part (where) and Storage Form (how). To our best knowledge, it is
the first survey that systematically studies the methodologies which make use
of various historical statistics when training deep neural networks. The
discussions with related topics like recurrent/memory networks, ensemble
learning, and reinforcement learning are demonstrated. We also expose future
challenges of this topic and encourage the community to pay attention to the
think of historical learning principles when designing algorithms. The paper
list related to historical learning is available at
\url{https://github.com/Martinser/Awesome-Historical-Learning.}
