  Spiking Neural Networks (SNN). SNNs are based on a more biologically inspired
approach than usual artificial neural networks. Such models are characterized
by complex dynamics between neurons and spikes. These are very sensitive to the
hyperparameters, making their optimization challenging. To tackle
hyperparameter optimization of SNNs, we initially extended the signal loss
issue of SNNs to what we call silent networks. These networks fail to emit
enough spikes at their outputs due to mistuned hyperparameters or architecture.
Generally, search spaces are heavily restrained, sometimes even discretized, to
prevent the sampling of such networks. By defining an early stopping criterion
detecting silent networks and by designing specific constraints, we were able
to instantiate larger and more flexible search spaces. We applied a constrained
Bayesian optimization technique, which was asynchronously parallelized, as the
evaluation time of a SNN is highly stochastic. Large-scale experiments were
carried-out on a multi-GPU Petascale architecture. By leveraging silent
networks, results show an acceleration of the search, while maintaining good
performances of both the optimization algorithm and the best solution obtained.
We were able to apply our methodology to two popular training algorithms, known
as spike timing dependent plasticity and surrogate gradient. Early detection
allowed us to prevent worthless and costly computation, directing the search
toward promising hyperparameter combinations. Our methodology could be applied
to multi-objective problems, where the spiking activity is often minimized to
reduce the energy consumption. In this scenario, it becomes essential to find
the delicate frontier between low-spiking and silent networks. Finally, our
approach may have implications for neural architecture search, particularly in
defining suitable spiking architectures.
