  In visual-based Reinforcement Learning (RL), agents often struggle to
generalize well to environmental variations in the state space that were not
observed during training. The variations can arise in both task-irrelevant
features, such as background noise, and task-relevant features, such as robot
configurations, that are related to the optimal decisions. To achieve
generalization in both situations, agents are required to accurately understand
the impact of changed features on the decisions, i.e., establishing the true
associations between changed features and decisions in the policy model.
However, due to the inherent correlations among features in the state space,
the associations between features and decisions become entangled, making it
difficult for the policy to distinguish them. To this end, we propose
Saliency-Guided Features Decorrelation (SGFD) to eliminate these correlations
through sample reweighting. Concretely, SGFD consists of two core techniques:
Random Fourier Functions (RFF) and the saliency map. RFF is utilized to
estimate the complex non-linear correlations in high-dimensional images, while
the saliency map is designed to identify the changed features. Under the
guidance of the saliency map, SGFD employs sample reweighting to minimize the
estimated correlations related to changed features, thereby achieving
decorrelation in visual RL tasks. Our experimental results demonstrate that
SGFD can generalize well on a wide range of test environments and significantly
outperforms state-of-the-art methods in handling both task-irrelevant
variations and task-relevant variations.
