  This report will show the history of deep learning evolves. It will trace
back as far as the initial belief of connectionism modelling of brain, and come
back to look at its early stage realization: neural networks. With the
background of neural network, we will gradually introduce how convolutional
neural network, as a representative of deep discriminative models, is developed
from neural networks, together with many practical techniques that can help in
optimization of neural networks. On the other hand, we will also trace back to
see the evolution history of deep generative models, to see how researchers
balance the representation power and computation complexity to reach Restricted
Boltzmann Machine and eventually reach Deep Belief Nets. Further, we will also
look into the development history of modelling time series data with neural
networks. We start with Time Delay Neural Networks and move further to
currently famous model named Recurrent Neural Network and its extension Long
Short Term Memory. We will also briefly look into how to construct deep
recurrent neural networks. Finally, we will conclude this report with some
interesting open-ended questions of deep neural networks.
