  Optimization for deep networks is currently a very active area of research.
As neural networks become deeper, the ability in manually optimizing the
network becomes harder. Mini-batch normalization, identification of effective
respective fields, momentum updates, introduction of residual blocks, learning
rate adoption, etc. have been proposed to speed up the rate of convergent in
manual training process while keeping the higher accuracy level. However, the
problem of finding optimal topological structure for a given problem is
becoming a challenging task need to be addressed immediately. Few researchers
have attempted to optimize the network structure using evolutionary computing
approaches. Among them, few have successfully evolved networks with
reinforcement learning and long-short-term memory. A very few has applied
evolutionary programming into deep convolution neural networks. These attempts
are mainly evolved the network structure and then subsequently optimized the
hyper-parameters of the network. However, a mechanism to evolve the deep
network structure under the techniques currently being practiced in manual
process is still absent. Incorporation of such techniques into chromosomes
level of evolutionary computing, certainly can take us to better topological
deep structures. The paper concludes by identifying the gap between
evolutionary based deep neural networks and deep neural networks. Further, it
proposes some insights for optimizing deep neural networks using evolutionary
computing techniques.
