  Deep learning is a topic of considerable current interest. The availability
of massive data collections and powerful software resources has led to an
impressive amount of results in many application areas that reveal essential
but hidden properties of the observations. System identification learns
mathematical descriptions of dynamic systems from input-output data and can
thus benefit from the advances of deep neural networks to enrich the possible
range of models to choose from. For this reason, we provide a survey of deep
learning from a system identification perspective. We cover a wide spectrum of
topics to enable researchers to understand the methods, providing rigorous
practical and theoretical insights into the benefits and challenges of using
them. The main aim of the identified model is to predict new data from previous
observations. This can be achieved with different deep learning based modelling
techniques and we discuss architectures commonly adopted in the literature,
like feedforward, convolutional, and recurrent networks. Their parameters have
to be estimated from past data trying to optimize the prediction performance.
For this purpose, we discuss a specific set of first-order optimization tools
that is emerged as efficient. The survey then draws connections to the
well-studied area of kernel-based methods. They control the data fit by
regularization terms that penalize models not in line with prior assumptions.
We illustrate how to cast them in deep architectures to obtain deep
kernel-based methods. The success of deep learning also resulted in surprising
empirical observations, like the counter-intuitive behaviour of models with
many parameters. We discuss the role of overparameterized models, including
their connection to kernels, as well as implicit regularization mechanisms
which affect generalization, specifically the interesting phenomena of benign
overfitting ...
