  Deep learning involves a difficult non-convex optimization problem, which is
often solved by stochastic gradient (SG) methods. While SG is usually
effective, it may not be robust in some situations. Recently, Newton methods
have been investigated as an alternative optimization technique, but nearly all
existing studies consider only fully-connected feedforward neural networks.
They do not investigate other types of networks such as Convolutional Neural
Networks (CNN), which are more commonly used in deep-learning applications. One
reason is that Newton methods for CNN involve complicated operations, and so
far no works have conducted a thorough investigation. In this work, we give
details of all building blocks including function, gradient, and Jacobian
evaluation, and Gauss-Newton matrix-vector products. These basic components are
very important because with them further developments of Newton methods for CNN
become possible. We show that an efficient MATLAB implementation can be done in
just several hundred lines of code and demonstrate that the Newton method gives
competitive test accuracy.
