  Gradient descent training techniques are remarkably successful in training
analog-valued artificial neural networks (ANNs). Such training techniques,
however, do not transfer easily to spiking networks due to the spike generation
hard non-linearity and the discrete nature of spike communication. We show that
in a feedforward spiking network that uses a temporal coding scheme where
information is encoded in spike times instead of spike rates, the network
input-output relation is differentiable almost everywhere. Moreover, this
relation is piece-wise linear after a transformation of variables. Methods for
training ANNs thus carry directly to the training of such spiking networks as
we show when training on the permutation invariant MNIST task. In contrast to
rate-based spiking networks that are often used to approximate the behavior of
ANNs, the networks we present spike much more sparsely and their behavior can
not be directly approximated by conventional ANNs. Our results highlight a new
approach for controlling the behavior of spiking networks with realistic
temporal dynamics, opening up the potential for using these networks to process
spike patterns with complex temporal information.
