  In this chapter we review the main literature related to the recent
advancement of deep neural-kernel architecture, an approach that seek the
synergy between two powerful class of models, i.e. kernel-based models and
artificial neural networks. The introduced deep neural-kernel framework is
composed of a hybridization of the neural networks architecture and a kernel
machine. More precisely, for the kernel counterpart the model is based on Least
Squares Support Vector Machines with explicit feature mapping. Here we discuss
the use of one form of an explicit feature map obtained by random Fourier
features. Thanks to this explicit feature map, in one hand bridging the two
architectures has become more straightforward and on the other hand one can
find the solution of the associated optimization problem in the primal,
therefore making the model scalable to large scale datasets. We begin by
introducing a neural-kernel architecture that serves as the core module for
deeper models equipped with different pooling layers. In particular, we review
three neural-kernel machines with average, maxout and convolutional pooling
layers. In average pooling layer the outputs of the previous representation
layers are averaged. The maxout layer triggers competition among different
input representations and allows the formation of multiple sub-networks within
the same model. The convolutional pooling layer reduces the dimensionality of
the multi-scale output representations. Comparison with neural-kernel model,
kernel based models and the classical neural networks architecture have been
made and the numerical experiments illustrate the effectiveness of the
introduced models on several benchmark datasets.
