  Offline reinforcement learning (RL) allows agents to learn effective,
return-maximizing policies from a static dataset. Three popular algorithms for
offline RL are Conservative Q-Learning (CQL), Behavior Cloning (BC), and
Decision Transformer (DT), from the class of Q-Learning, Imitation Learning,
and Sequence Modeling respectively. A key open question is: which algorithm is
preferred under what conditions? We study this question empirically by
exploring the performance of these algorithms across the commonly used D4RL and
Robomimic benchmarks. We design targeted experiments to understand their
behavior concerning data suboptimality, task complexity, and stochasticity. Our
key findings are: (1) DT requires more data than CQL to learn competitive
policies but is more robust; (2) DT is a substantially better choice than both
CQL and BC in sparse-reward and low-quality data settings; (3) DT and BC are
preferable as task horizon increases, or when data is obtained from human
demonstrators; and (4) CQL excels in situations characterized by the
combination of high stochasticity and low data quality. We also investigate
architectural choices and scaling trends for DT on Atari and D4RL and make
design/scaling recommendations. We find that scaling the amount of data for DT
by 5x gives a 2.5x average score improvement on Atari.
