  Offline reinforcement learning (RL) is a promising direction that allows RL
agents to pre-train on large datasets, avoiding the recurrence of expensive
data collection. To advance the field, it is crucial to generate large-scale
datasets. Compositional RL is particularly appealing for generating such large
datasets, since 1)~it permits creating many tasks from few components, 2)~the
task structure may enable trained agents to solve new tasks by combining
relevant learned components, and 3)~the compositional dimensions provide a
notion of task relatedness. This paper provides four offline RL datasets for
simulated robotic manipulation created using the $256$ tasks from CompoSuite
[Mendez at al., 2022a]. Each dataset is collected from an agent with a
different degree of performance, and consists of $256$ million transitions. We
provide training and evaluation settings for assessing an agent's ability to
learn compositional task policies. Our benchmarking experiments show that
current offline RL methods can learn the training tasks to some extent and that
compositional methods outperform non-compositional methods. Yet current methods
are unable to extract the compositional structure to generalize to unseen
tasks, highlighting a need for future research in offline compositional RL.
