  Although statistical learning theory provides a robust framework to
understand supervised learning, many theoretical aspects of deep learning
remain unclear, in particular how different architectures may lead to inductive
bias when trained using gradient based methods. The goal of these lectures is
to provide an overview of some of the main questions that arise when attempting
to understand deep learning from a learning theory perspective. After a brief
reminder on statistical learning theory and stochastic optimization, we discuss
implicit bias in the context of benign overfitting. We then move to a general
description of the mirror descent algorithm, showing how we may go back and
forth between a parameter space and the corresponding function space for a
given learning problem, as well as how the geometry of the learning problem may
be represented by a metric tensor. Building on this framework, we provide a
detailed study of the implicit bias of gradient descent on linear diagonal
networks for various regression tasks, showing how the loss function, scale of
parameters at initialization and depth of the network may lead to various forms
of implicit bias, in particular transitioning between kernel or feature
learning.
