  Offline reinforcement learning agents seek optimal policies from fixed data
sets. With environmental interaction prohibited, agents face significant
challenges in preventing errors in value estimates from compounding and
subsequently causing the learning process to collapse. Uncertainty estimation
using ensembles compensates for this by penalising high-variance value
estimates, allowing agents to learn robust policies based on data-driven
actions. However, the requirement for large ensembles to facilitate sufficient
penalisation results in significant computational overhead. In this work, we
examine the role of policy constraints as a mechanism for regulating
uncertainty, and the corresponding balance between level of constraint and
ensemble size. By incorporating behavioural cloning into policy updates, we
show empirically that sufficient penalisation can be achieved with a much
smaller ensemble size, substantially reducing computational demand while
retaining state-of-the-art performance on benchmarking tasks. Furthermore, we
show how such an approach can facilitate stable online fine tuning, allowing
for continued policy improvement while avoiding severe performance drops.
