  Offline reinforcement learning (RL) has attracted much attention due to its
ability in learning from static offline datasets and eliminating the need of
interacting with the environment. Nevertheless, the success of offline RL
relies heavily on the offline transitions annotated with reward labels. In
practice, we often need to hand-craft the reward function, which is sometimes
difficult, labor-intensive, or inefficient. To tackle this challenge, we set
our focus on the offline imitation learning (IL) setting, and aim at getting a
reward function based on the expert data and unlabeled data. To that end, we
propose a simple yet effective search-based offline IL method, tagged SEABO.
SEABO allocates a larger reward to the transition that is close to its closest
neighbor in the expert demonstration, and a smaller reward otherwise, all in an
unsupervised learning manner. Experimental results on a variety of D4RL
datasets indicate that SEABO can achieve competitive performance to offline RL
algorithms with ground-truth rewards, given only a single expert trajectory,
and can outperform prior reward learning and offline IL methods across many
tasks. Moreover, we demonstrate that SEABO also works well if the expert
demonstrations contain only observations. Our code is publicly available at
https://github.com/dmksjfl/SEABO.
