  This work aims to tackle a major challenge in offline Inverse Reinforcement
Learning (IRL), namely the reward extrapolation error, where the learned reward
function may fail to explain the task correctly and misguide the agent in
unseen environments due to the intrinsic covariate shift. Leveraging both
expert data and lower-quality diverse data, we devise a principled algorithm
(namely CLARE) that solves offline IRL efficiently via integrating
"conservatism" into a learned reward function and utilizing an estimated
dynamics model. Our theoretical analysis provides an upper bound on the return
gap between the learned policy and the expert policy, based on which we
characterize the impact of covariate shift by examining subtle two-tier
tradeoffs between the exploitation (on both expert and diverse data) and
exploration (on the estimated dynamics model). We show that CLARE can provably
alleviate the reward extrapolation error by striking the right
exploitation-exploration balance therein. Extensive experiments corroborate the
significant performance gains of CLARE over existing state-of-the-art
algorithms on MuJoCo continuous control tasks (especially with a small offline
dataset), and the learned reward is highly instructive for further learning.
