  Existing model-based value expansion methods typically leverage a world model
for value estimation with a fixed rollout horizon to assist policy learning.
However, the fixed rollout with an inaccurate model has a potential to harm the
learning process. In this paper, we investigate the idea of using the model
knowledge for value expansion adaptively. We propose a novel method called
Dynamic-horizon Model-based Value Expansion (DMVE) to adjust the world model
usage with different rollout horizons. Inspired by reconstruction-based
techniques that can be applied for visual data novelty detection, we utilize a
world model with a reconstruction module for image feature extraction, in order
to acquire more precise value estimation. The raw and the reconstructed images
are both used to determine the appropriate horizon for adaptive value
expansion. On several benchmark visual control tasks, experimental results show
that DMVE outperforms all baselines in sample efficiency and final performance,
indicating that DMVE can achieve more effective and accurate value estimation
than state-of-the-art model-based methods.
