  Equipped with the trained environmental dynamics, model-based offline
reinforcement learning (RL) algorithms can often successfully learn good
policies from fixed-sized datasets, even some datasets with poor quality.
Unfortunately, however, it can not be guaranteed that the generated samples
from the trained dynamics model are reliable (e.g., some synthetic samples may
lie outside of the support region of the static dataset). To address this
issue, we propose Trajectory Truncation with Uncertainty (TATU), which
adaptively truncates the synthetic trajectory if the accumulated uncertainty
along the trajectory is too large. We theoretically show the performance bound
of TATU to justify its benefits. To empirically show the advantages of TATU, we
first combine it with two classical model-based offline RL algorithms, MOPO and
COMBO. Furthermore, we integrate TATU with several off-the-shelf model-free
offline RL algorithms, e.g., BCQ. Experimental results on the D4RL benchmark
show that TATU significantly improves their performance, often by a large
margin. Code is available here.
