  How to improve the ability of scene representation is a key issue in
vision-oriented decision-making applications, and current approaches usually
learn task-relevant state representations within visual reinforcement learning
to address this problem. While prior work typically introduces one-step
behavioral similarity metrics with elements (e.g., rewards and actions) to
extract task-relevant state information from observations, they often ignore
the inherent dynamics relationships among the elements that are essential for
learning accurate representations, which further impedes the discrimination of
short-term similar task/behavior information in long-term dynamics transitions.
To alleviate this problem, we propose an intrinsic dynamics-driven
representation learning method with sequence models in visual reinforcement
learning, namely DSR. Concretely, DSR optimizes the parameterized encoder by
the state-transition dynamics of the underlying system, which prompts the
latent encoding information to satisfy the state-transition process and then
the state space and the noise space can be distinguished. In the implementation
and to further improve the representation ability of DSR on encoding similar
tasks, sequential elements' frequency domain and multi-step prediction are
adopted for sequentially modeling the inherent dynamics. Finally, experimental
results show that DSR has achieved significant performance improvements in the
visual Distracting DMControl control tasks, especially with an average of
78.9\% over the backbone baseline. Further results indicate that it also
achieves the best performances in real-world autonomous driving applications on
the CARLA simulator. Moreover, qualitative analysis results validate that our
method possesses the superior ability to learn generalizable scene
representations on visual tasks. The source code is available at
https://github.com/DMU-XMU/DSR.
