  In this paper, we present \textbf{C}ont\textbf{E}xtual \textbf{I}mitation
\textbf{L}earning~(CEIL), a general and broadly applicable algorithm for
imitation learning (IL). Inspired by the formulation of hindsight information
matching, we derive CEIL by explicitly learning a hindsight embedding function
together with a contextual policy using the hindsight embeddings. To achieve
the expert matching objective for IL, we advocate for optimizing a contextual
variable such that it biases the contextual policy towards mimicking expert
behaviors. Beyond the typical learning from demonstrations (LfD) setting, CEIL
is a generalist that can be effectively applied to multiple settings including:
1)~learning from observations (LfO), 2)~offline IL, 3)~cross-domain IL
(mismatched experts), and 4) one-shot IL settings. Empirically, we evaluate
CEIL on the popular MuJoCo tasks (online) and the D4RL dataset (offline).
Compared to prior state-of-the-art baselines, we show that CEIL is more
sample-efficient in most online IL tasks and achieves better or competitive
performances in offline tasks.
