  The Laplacian representation recently gains increasing attention for
reinforcement learning as it provides succinct and informative representation
for states, by taking the eigenvectors of the Laplacian matrix of the
state-transition graph as state embeddings. Such representation captures the
geometry of the underlying state space and is beneficial to RL tasks such as
option discovery and reward shaping. To approximate the Laplacian
representation in large (or even continuous) state spaces, recent works propose
to minimize a spectral graph drawing objective, which however has infinitely
many global minimizers other than the eigenvectors. As a result, their learned
Laplacian representation may differ from the ground truth. To solve this
problem, we reformulate the graph drawing objective into a generalized form and
derive a new learning objective, which is proved to have eigenvectors as its
unique global minimizer. It enables learning high-quality Laplacian
representations that faithfully approximate the ground truth. We validate this
via comprehensive experiments on a set of gridworld and continuous control
environments. Moreover, we show that our learned Laplacian representations lead
to more exploratory options and better reward shaping.
