  Recent years have witnessed significant advancements in offline reinforcement
learning (RL), resulting in the development of numerous algorithms with varying
degrees of complexity. While these algorithms have led to noteworthy
improvements, many incorporate seemingly minor design choices that impact their
effectiveness beyond core algorithmic advances. However, the effect of these
design choices on established baselines remains understudied. In this work, we
aim to bridge this gap by conducting a retrospective analysis of recent works
in offline RL and propose ReBRAC, a minimalistic algorithm that integrates such
design elements built on top of the TD3+BC method. We evaluate ReBRAC on 51
datasets with both proprioceptive and visual state spaces using D4RL and V-D4RL
benchmarks, demonstrating its state-of-the-art performance among ensemble-free
methods in both offline and offline-to-online settings. To further illustrate
the efficacy of these design choices, we perform a large-scale ablation study
and hyperparameter sensitivity analysis on the scale of thousands of
experiments.
