  The recent development of reinforcement learning (RL) has boosted the
adoption of online RL for wireless radio resource management (RRM). However,
online RL algorithms require direct interactions with the environment, which
may be undesirable given the potential performance loss due to the unavoidable
exploration in RL. In this work, we first investigate the use of \emph{offline}
RL algorithms in solving the RRM problem. We evaluate several state-of-the-art
offline RL algorithms, including behavior constrained Q-learning (BCQ),
conservative Q-learning (CQL), and implicit Q-learning (IQL), for a specific
RRM problem that aims at maximizing a linear combination {of sum and}
5-percentile rates via user scheduling. We observe that the performance of
offline RL for the RRM problem depends critically on the behavior policy used
for data collection, and further propose a novel offline RL solution that
leverages heterogeneous datasets collected by different behavior policies. We
show that with a proper mixture of the datasets, offline RL can produce a
near-optimal RL policy even when all involved behavior policies are highly
suboptimal.
