  Tackling new machine learning problems with neural networks always means
optimizing numerous hyperparameters that define their structure and strongly
impact their performances. In this work, we study the use of goal-oriented
sensitivity analysis, based on the Hilbert-Schmidt Independence Criterion
(HSIC), for hyperparameter analysis and optimization. Hyperparameters live in
spaces that are often complex and awkward. They can be of different natures
(categorical, discrete, boolean, continuous), interact, and have
inter-dependencies. All this makes it non-trivial to perform classical
sensitivity analysis. We alleviate these difficulties to obtain a robust
analysis index that is able to quantify hyperparameters' relative impact on a
neural network's final error. This valuable tool allows us to better understand
hyperparameters and to make hyperparameter optimization more interpretable. We
illustrate the benefits of this knowledge in the context of hyperparameter
optimization and derive an HSIC-based optimization algorithm that we apply on
MNIST and Cifar, classical machine learning data sets, but also on the
approximation of Runge function and Bateman equations solution, of interest for
scientific machine learning. This method yields neural networks that are both
competitive and cost-effective.
