  Simulation is used extensively in autonomous systems, particularly in robotic
manipulation. By far, the most common approach is to train a controller in
simulation, and then use it as an initial starting point for the real system.
We demonstrate how to learn simultaneously from both simulation and interaction
with the real environment. We propose an algorithm for balancing the large
number of samples from the high throughput but less accurate simulation and the
low-throughput, high-fidelity and costly samples from the real environment. We
achieve that by maintaining a replay buffer for each environment the agent
interacts with. We analyze such multi-environment interaction theoretically,
and provide convergence properties, through a novel theoretical replay buffer
analysis. We demonstrate the efficacy of our method on a sim-to-real
environment.
