  We present a reinforcement learning-based solution to autonomously race on a
miniature race car platform. We show that a policy that is trained purely in
simulation using a relatively simple vehicle model, including model
randomization, can be successfully transferred to the real robotic setup. We
achieve this by using novel policy output regularization approach and a lifted
action space which enables smooth actions but still aggressive race car
driving. We show that this regularized policy does outperform the Soft Actor
Critic (SAC) baseline method, both in simulation and on the real car, but it is
still outperformed by a Model Predictive Controller (MPC) state of the art
method. The refinement of the policy with three hours of real-world interaction
data allows the reinforcement learning policy to achieve lap times similar to
the MPC controller while reducing track constraint violations by 50%.
