  Neural networks, specifically deep convolutional neural networks, have
achieved unprecedented performance in various computer vision tasks, but the
rationale for the computations and structures of successful neural networks is
not fully understood. Theories abound for the aptitude of convolutional neural
networks for image classification, but less is understood about why such models
would be capable of complex visual tasks such as inference and anomaly
identification. Here, we propose a sparse coding interpretation of neural
networks that have ReLU activation and of convolutional neural networks in
particular. In sparse coding, when the model's basis functions are assumed to
be orthogonal, the optimal coefficients are given by the soft-threshold
function of the basis functions projected onto the input image. In a
non-negative variant of sparse coding, the soft-threshold function becomes a
ReLU. Here, we derive these solutions via sparse coding with orthogonal-assumed
basis functions, then we derive the convolutional neural network forward
transformation from a modified non-negative orthogonal sparse coding model with
an exponential prior parameter for each sparse coding coefficient. Next, we
derive a complete convolutional neural network without normalization and
pooling by adding logistic regression to a hierarchical sparse coding model.
Finally we motivate potentially more robust forward transformations by
maintaining sparse priors in convolutional neural networks as well performing a
stronger nonlinear transformation.
