  In this paper, we consider reinforcement learning of nonlinear systems with
continuous state and action spaces. We present an episodic learning algorithm,
where we for each episode use convex optimization to find a two-layer neural
network approximation of the optimal $Q$-function. The convex optimization
approach guarantees that the weights calculated at each episode are optimal,
with respect to the given sampled states and actions of the current episode.
For stable nonlinear systems, we show that the algorithm converges and that the
converging parameters of the trained neural network can be made arbitrarily
close to the optimal neural network parameters. In particular, if the
regularization parameter in the training phase is given by $\rho$, then the
parameters of the trained neural network converge to $w$, where the distance
between $w$ and the optimal parameters $w^\star$ is bounded by
$\mathcal{O}(\rho)$. That is, when the number of episodes goes to infinity,
there exists a constant $C$ such that \[
  \|w-w^\star\| \le C\rho. \]
  In particular, our algorithm converges arbitrarily close to the optimal
neural network parameters as the regularization parameter goes to zero. As a
consequence, our algorithm converges fast due to the polynomial-time
convergence of convex optimization algorithms.
