  Conditional Imitation learning is a common and effective approach to train
autonomous driving agents. However, two issues limit the full potential of this
approach: (i) the inertia problem, a special case of causal confusion where the
agent mistakenly correlates low speed with no acceleration, and (ii) low
correlation between offline and online performance due to the accumulation of
small errors that brings the agent in a previously unseen state. Both issues
are critical for state-aware models, yet informing the driving agent of its
internal state as well as the state of the environment is of crucial
importance. In this paper we propose a multi-task learning agent based on a
multi-stage vision transformer with state token propagation. We feed the state
of the vehicle along with the representation of the environment as a special
token of the transformer and propagate it throughout the network. This allows
us to tackle the aforementioned issues from different angles: guiding the
driving policy with learned stop/go information, performing data augmentation
directly on the state of the vehicle and visually explaining the model's
decisions. We report a drastic decrease in inertia and a high correlation
between offline and online metrics.
