  Reinforcement Learning (RL) has the potential to surpass human performance in
driving without needing any expert supervision. Despite its promise, the
state-of-the-art in sensorimotor self-driving is dominated by imitation
learning methods due to the inherent shortcomings of RL algorithms.
Nonetheless, RL agents are able to discover highly successful policies when
provided with privileged ground truth representations of the environment. In
this work, we investigate what separates privileged RL agents from sensorimotor
agents for urban driving in order to bridge the gap between the two. We propose
vision-based deep learning models to approximate the privileged representations
from sensor data. In particular, we identify aspects of state representation
that are crucial for the success of the RL agent such as desired route
generation and stop zone prediction, and propose solutions to gradually develop
less privileged RL agents. We also observe that bird's-eye-view models trained
on offline datasets do not generalize to online RL training due to distribution
mismatch. Through rigorous evaluation on the CARLA simulation environment, we
shed light on the significance of the state representations in RL for
autonomous driving and point to unresolved challenges for future research.
