  Deep Reinforcement Learning (RL) powered by neural net approximation of the Q
function has had enormous empirical success. While the theory of RL has
traditionally focused on linear function approximation (or eluder dimension)
approaches, little is known about nonlinear RL with neural net approximations
of the Q functions. This is the focus of this work, where we study function
approximation with two-layer neural networks (considering both ReLU and
polynomial activation functions). Our first result is a computationally and
statistically efficient algorithm in the generative model setting under
completeness for two-layer neural networks. Our second result considers this
setting but under only realizability of the neural net function class. Here,
assuming deterministic dynamics, the sample complexity scales linearly in the
algebraic dimension. In all cases, our results significantly improve upon what
can be attained with linear (or eluder dimension) methods.
