  Despite the rapid growth in datasets for video activity, stable robust
activity recognition with neural networks remains challenging. This is in large
part due to the explosion of possible variation in video -- including lighting
changes, object variation, movement variation, and changes in surrounding
context. An alternative is to make use of simulation data, where all of these
factors can be artificially controlled. In this paper, we propose the
Randomized Simulation as Augmentation (RSA) framework which augments real-world
training data with synthetic data to improve the robustness of action
recognition networks. We generate large-scale synthetic datasets with
randomized nuisance factors. We show that training with such extra data, when
appropriately constrained, can significantly improve the performance of the
state-of-the-art I3D networks or, conversely, reduce the number of labeled real
videos needed to achieve good performance. Experiments on two real-world
datasets NTU RGB+D and VIRAT demonstrate the effectiveness of our method.
