  We propose a simple architecture for deep reinforcement learning by embedding
inputs into a learned Fourier basis and show that it improves the sample
efficiency of both state-based and image-based RL. We perform infinite-width
analysis of our architecture using the Neural Tangent Kernel and theoretically
show that tuning the initial variance of the Fourier basis is equivalent to
functional regularization of the learned deep network. That is, these learned
Fourier features allow for adjusting the degree to which networks underfit or
overfit different frequencies in the training data, and hence provide a
controlled mechanism to improve the stability and performance of RL
optimization. Empirically, this allows us to prioritize learning low-frequency
functions and speed up learning by reducing networks' susceptibility to noise
in the optimization process, such as during Bellman updates. Experiments on
standard state-based and image-based RL benchmarks show clear benefits of our
architecture over the baselines. Website at
https://alexanderli.com/learned-fourier-features
