  Recent works have derived neural networks with online correlation-based
learning rules to perform \textit{kernel similarity matching}. These works
applied existing linear similarity matching algorithms to nonlinear features
generated with random Fourier methods. In this paper attempt to perform kernel
similarity matching by directly learning the nonlinear features. Our algorithm
proceeds by deriving and then minimizing an upper bound for the sum of squared
errors between output and input kernel similarities. The construction of our
upper bound leads to online correlation-based learning rules which can be
implemented with a 1 layer recurrent neural network. In addition to generating
high-dimensional linearly separable representations, we show that our upper
bound naturally yields representations which are sparse and selective for
specific input patterns. We compare the approximation quality of our method to
neural random Fourier method and variants of the popular but non-biological
"Nystr{\"o}m" method for approximating the kernel matrix. Our method appears to
be comparable or better than randomly sampled Nystr{\"o}m methods when the
outputs are relatively low dimensional (although still potentially higher
dimensional than the inputs) but less faithful when the outputs are very high
dimensional.
