  Spiking Neural Networks (SNNs) have the potential for rich spatio-temporal
signal processing thanks to exploiting both spatial and temporal parameters.
The temporal dynamics such as time constants of the synapses and neurons and
delays have been recently shown to have computational benefits that help reduce
the overall number of parameters required in the network and increase the
accuracy of the SNNs in solving temporal tasks. Optimizing such temporal
parameters, for example, through gradient descent, gives rise to a temporal
architecture for different problems. As has been shown in machine learning, to
reduce the cost of optimization, architectural biases can be applied, in this
case in the temporal domain. Such inductive biases in temporal parameters have
been found in neuroscience studies, highlighting a hierarchy of temporal
structure and input representation in different layers of the cortex. Motivated
by this, we propose to impose a hierarchy of temporal representation in the
hidden layers of SNNs, highlighting that such an inductive bias improves their
performance. We demonstrate the positive effects of temporal hierarchy in the
time constants of feed-forward SNNs applied to temporal tasks (Multi-Time-Scale
XOR and Keyword Spotting, with a benefit of up to 4.1% in classification
accuracy). Moreover, we show that such architectural biases, i.e. hierarchy of
time constants, naturally emerge when optimizing the time constants through
gradient descent, initialized as homogeneous values. We further pursue this
proposal in temporal convolutional SNNs, by introducing the hierarchical bias
in the size and dilation of temporal kernels, giving rise to competitive
results in popular temporal spike-based datasets.
