  Offline inverse reinforcement learning (Offline IRL) aims to recover the
structure of rewards and environment dynamics that underlie observed actions in
a fixed, finite set of demonstrations from an expert agent. Accurate models of
expertise in executing a task has applications in safety-sensitive applications
such as clinical decision making and autonomous driving. However, the structure
of an expert's preferences implicit in observed actions is closely linked to
the expert's model of the environment dynamics (i.e. the ``world'' model).
Thus, inaccurate models of the world obtained from finite data with limited
coverage could compound inaccuracy in estimated rewards. To address this issue,
we propose a bi-level optimization formulation of the estimation task wherein
the upper level is likelihood maximization based upon a conservative model of
the expert's policy (lower level). The policy model is conservative in that it
maximizes reward subject to a penalty that is increasing in the uncertainty of
the estimated model of the world. We propose a new algorithmic framework to
solve the bi-level optimization problem formulation and provide statistical and
computational guarantees of performance for the associated optimal reward
estimator. Finally, we demonstrate that the proposed algorithm outperforms the
state-of-the-art offline IRL and imitation learning benchmarks by a large
margin, over the continuous control tasks in MuJoCo and different datasets in
the D4RL benchmark.
