  Deep neural networks (NN) have achieved great success in many applications.
However, why do deep neural networks obtain good generalization at an
over-parameterization regime is still unclear. To better understand deep NN, we
establish the connection between deep NN and a novel kernel family, i.e.,
Neural Optimization Kernel (NOK). The architecture of structured approximation
of NOK performs monotonic descent updates of implicit regularization problems.
We can implicitly choose the regularization problems by employing different
activation functions, e.g., ReLU, max pooling, and soft-thresholding. We
further establish a new generalization bound of our deep structured
approximated NOK architecture. Our unsupervised structured approximated NOK
block can serve as a simple plug-in of popular backbones for a good
generalization against input noise.
