  Behavioural cloning (BC) is a commonly used imitation learning method to
infer a sequential decision-making policy from expert demonstrations. However,
when the quality of the data is not optimal, the resulting behavioural policy
also performs sub-optimally once deployed. Recently, there has been a surge in
offline reinforcement learning methods that hold the promise to extract
high-quality policies from sub-optimal historical data. A common approach is to
perform regularisation during training, encouraging updates during policy
evaluation and/or policy improvement to stay close to the underlying data. In
this work, we investigate whether an offline approach to improving the quality
of the existing data can lead to improved behavioural policies without any
changes in the BC algorithm. The proposed data improvement approach -
Trajectory Stitching (TS) - generates new trajectories (sequences of states and
actions) by `stitching' pairs of states that were disconnected in the original
data and generating their connecting new action. By construction, these new
transitions are guaranteed to be highly plausible according to probabilistic
models of the environment, and to improve a state-value function. We
demonstrate that the iterative process of replacing old trajectories with new
ones incrementally improves the underlying behavioural policy. Extensive
experimental results show that significant performance gains can be achieved
using TS over BC policies extracted from the original data. Furthermore, using
the D4RL benchmarking suite, we demonstrate that state-of-the-art results are
obtained by combining TS with two existing offline learning methodologies
reliant on BC, model-based offline planning (MBOP) and policy constraint
(TD3+BC).
