  Recent work has focused on combining kernel methods and deep learning to
exploit the best of the two approaches. Here, we introduce a new architecture
of neural networks in which we replace the top dense layers of standard
convolutional architectures with an approximation of a kernel function by
relying on the Nystr{\"o}m approximation. Our approach is easy and highly
flexible. It is compatible with any kernel function and it allows exploiting
multiple kernels. We show that our architecture has the same performance than
standard architecture on datasets like SVHN and CIFAR100. One benefit of the
method lies in its limited number of learnable parameters which makes it
particularly suited for small training set sizes, e.g. from 5 to 20 samples per
class.
