  We study a new paradigm for sequential decision making, called offline policy
learning from observations (PLfO). Offline PLfO aims to learn policies using
datasets with substandard qualities: 1) only a subset of trajectories is
labeled with rewards, 2) labeled trajectories may not contain actions, 3)
labeled trajectories may not be of high quality, and 4) the data may not have
full coverage. Such imperfection is common in real-world learning scenarios,
and offline PLfO encompasses many existing offline learning setups, including
offline imitation learning (IL), offline IL from observations (ILfO), and
offline reinforcement learning (RL). In this work, we present a generic
approach to offline PLfO, called $\textbf{M}$odality-agnostic
$\textbf{A}$dversarial $\textbf{H}$ypothesis $\textbf{A}$daptation for
$\textbf{L}$earning from $\textbf{O}$bservations (MAHALO). Built upon the
pessimism concept in offline RL, MAHALO optimizes the policy using a
performance lower bound that accounts for uncertainty due to the dataset's
insufficient coverage. We implement this idea by adversarially training
data-consistent critic and reward functions, which forces the learned policy to
be robust to data deficiency. We show that MAHALO consistently outperforms or
matches specialized algorithms across a variety of offline PLfO tasks in theory
and experiments. Our code is available at https://github.com/AnqiLi/mahalo.
