  Despite Deep Learning's (DL) empirical success, our theoretical understanding
of its efficacy remains limited. One notable paradox is that while conventional
wisdom discourages perfect data fitting, deep neural networks are designed to
do just that, yet they generalize effectively. This study focuses on exploring
this phenomenon attributed to the implicit bias at play. Various sources of
implicit bias have been identified, such as step size, weight initialization,
optimization algorithm, and number of parameters. In this work, we focus on
investigating the implicit bias originating from weight initialization. To this
end, we examine the problem of solving underdetermined linear systems in
various contexts, scrutinizing the impact of initialization on the implicit
regularization when using deep networks to solve such systems. Our findings
elucidate the role of initialization in the optimization and generalization
paradoxes, contributing to a more comprehensive understanding of DL's
performance characteristics.
