  Existing vision based supervised approaches to lateral vehicle control are
capable of directly mapping RGB images to the appropriate steering commands.
However, they are prone to suffering from inadequate robustness in real world
scenarios due to a lack of failure cases in the training data. In this paper, a
framework for training a more robust and scalable model for lateral vehicle
control is proposed. The framework only requires an unlabeled sequence of RGB
images. The trained model takes a point cloud as input and predicts the lateral
offset to a subsequent frame from which the steering angle is inferred. The
frame poses are in turn obtained from visual odometry. The point cloud is
conceived by projecting dense depth maps into 3D. An arbitrary number of
additional trajectories from this point cloud can be generated during training.
This is to increase the robustness of the model. Online experiments show that
the performance of our method is superior to that of the supervised model.
