  The gap between simulation and the real-world restrains many machine learning
breakthroughs in computer vision and reinforcement learning from being
applicable in the real world. In this work, we tackle this gap for the specific
case of camera-based navigation, formulating it as following a visual cue in
the foreground with arbitrary backgrounds. The visual cue in the foreground can
often be simulated realistically, such as a line, gate or cone. The challenge
then lies in coping with the unknown backgrounds and integrating both. As such,
the goal is to train a visual agent on data captured in an empty simulated
environment except for this foreground cue and test this model directly in a
visually diverse real world. In order to bridge this big gap, we show it's
crucial to combine following techniques namely: Randomized augmentation of the
fore- and background, regularization with both deep supervision and triplet
loss and finally abstraction of the dynamics by using waypoints rather than
direct velocity commands. The various techniques are ablated in our
experimental results both qualitatively and quantitatively finally
demonstrating a successful transfer from simulation to the real world.
