  When transferring a control policy from simulation to a physical system, the
policy needs to be robust to variations in the dynamics to perform well.
Commonly, the optimal policy overfits to the approximate model and the
corresponding state-distribution, often resulting in failure to trasnfer
underlying distributional shifts. In this paper, we present Robust Fitted Value
Iteration, which uses dynamic programming to compute the optimal value function
on the compact state domain and incorporates adversarial perturbations of the
system dynamics. The adversarial perturbations encourage a optimal policy that
is robust to changes in the dynamics. Utilizing the continuous-time perspective
of reinforcement learning, we derive the optimal perturbations for the states,
actions, observations and model parameters in closed-form. Notably, the
resulting algorithm does not require discretization of states or actions.
Therefore, the optimal adversarial perturbations can be efficiently
incorporated in the min-max value function update. We apply the resulting
algorithm to the physical Furuta pendulum and cartpole. By changing the masses
of the systems we evaluate the quantitative and qualitative performance across
different model parameters. We show that robust value iteration is more robust
compared to deep reinforcement learning algorithm and the non-robust version of
the algorithm. Videos of the experiments are shown at
https://sites.google.com/view/rfvi
