  Cross-Entropy Method (CEM) is commonly used for planning in model-based
reinforcement learning (MBRL) where a centralized approach is typically
utilized to update the sampling distribution based on only the top-$k$
operation's results on samples. In this paper, we show that such a centralized
approach makes CEM vulnerable to local optima, thus impairing its sample
efficiency. To tackle this issue, we propose Decentralized CEM (DecentCEM), a
simple but effective improvement over classical CEM, by using an ensemble of
CEM instances running independently from one another, and each performing a
local improvement of its own sampling distribution. We provide both theoretical
and empirical analysis to demonstrate the effectiveness of this simple
decentralized approach. We empirically show that, compared to the classical
centralized approach using either a single or even a mixture of Gaussian
distributions, our DecentCEM finds the global optimum much more consistently
thus improves the sample efficiency. Furthermore, we plug in our DecentCEM in
the planning problem of MBRL, and evaluate our approach in several continuous
control environments, with comparison to the state-of-art CEM based MBRL
approaches (PETS and POPLIN). Results show sample efficiency improvement by
simply replacing the classical CEM module with our DecentCEM module, while only
sacrificing a reasonable amount of computational cost. Lastly, we conduct
ablation studies for more in-depth analysis. Code is available at
https://github.com/vincentzhang/decentCEM
