  Offline reinforcement learning (RL) presents a promising approach for
learning reinforced policies from offline datasets without the need for costly
or unsafe interactions with the environment. However, datasets collected by
humans in real-world environments are often noisy and may even be maliciously
corrupted, which can significantly degrade the performance of offline RL. In
this work, we first investigate the performance of current offline RL
algorithms under comprehensive data corruption, including states, actions,
rewards, and dynamics. Our extensive experiments reveal that implicit
Q-learning (IQL) demonstrates remarkable resilience to data corruption among
various offline RL algorithms. Furthermore, we conduct both empirical and
theoretical analyses to understand IQL's robust performance, identifying its
supervised policy learning scheme as the key factor. Despite its relative
robustness, IQL still suffers from heavy-tail targets of Q functions under
dynamics corruption. To tackle this challenge, we draw inspiration from robust
statistics to employ the Huber loss to handle the heavy-tailedness and utilize
quantile estimators to balance penalization for corrupted data and learning
stability. By incorporating these simple yet effective modifications into IQL,
we propose a more robust offline RL approach named Robust IQL (RIQL). Extensive
experiments demonstrate that RIQL exhibits highly robust performance when
subjected to diverse data corruption scenarios.
