  We prove that training neural networks on 1-D data is equivalent to solving
convex Lasso problems with discrete, explicitly defined dictionary matrices. We
consider neural networks with piecewise linear activations and depths ranging
from 2 to an arbitrary but finite number of layers. We first show that
two-layer networks with piecewise linear activations are equivalent to Lasso
models using a discrete dictionary of ramp functions, with breakpoints
corresponding to the training data points. In certain general architectures
with absolute value or ReLU activations, a third layer surprisingly creates
features that reflect the training data about themselves. Additional layers
progressively generate reflections of these reflections. The Lasso
representation provides valuable insights into the analysis of globally optimal
networks, elucidating their solution landscapes and enabling closed-form
solutions in certain special cases. Numerical results show that reflections
also occur when optimizing standard deep networks using standard non-convex
optimizers. Additionally, we demonstrate our theory with autoregressive time
series models.
