  We uncover a surprising phenomenon in deep reinforcement learning: training a
diverse ensemble of data-sharing agents -- a well-established exploration
strategy -- can significantly impair the performance of the individual ensemble
members when compared to standard single-agent training. Through careful
analysis, we attribute the degradation in performance to the low proportion of
self-generated data in the shared training data for each ensemble member, as
well as the inefficiency of the individual ensemble members to learn from such
highly off-policy data. We thus name this phenomenon the curse of diversity. We
find that several intuitive solutions -- such as a larger replay buffer or a
smaller ensemble size -- either fail to consistently mitigate the performance
loss or undermine the advantages of ensembling. Finally, we demonstrate the
potential of representation learning to counteract the curse of diversity with
a novel method named Cross-Ensemble Representation Learning (CERL) in both
discrete and continuous control domains. Our work offers valuable insights into
an unexpected pitfall in ensemble-based exploration and raises important
caveats for future applications of similar approaches.
