  Deep neural networks are considered to be state of the art models in many
offline machine learning tasks. However, their performance and generalization
abilities in online learning tasks are much less understood. Therefore, we
focus on online learning and tackle the challenging problem where the
underlying process is stationary and ergodic and thus removing the i.i.d.
assumption and allowing observations to depend on each other arbitrarily. We
prove the generalization abilities of Lipschitz regularized deep neural
networks and show that by using those networks, a convergence to the best
possible prediction strategy is guaranteed.
