  Semi-gradient Q-learning is applied in many fields, but due to the absence of
an explicit loss function, studying its dynamics and implicit bias in the
parameter space is challenging. This paper introduces the Fokker--Planck
equation and employs partial data obtained through sampling to construct and
visualize the effective loss landscape within a two-dimensional parameter
space. This visualization reveals how the global minima in the loss landscape
can transform into saddle points in the effective loss landscape, as well as
the implicit bias of the semi-gradient method. Additionally, we demonstrate
that saddle points, originating from the global minima in loss landscape, still
exist in the effective loss landscape under high-dimensional parameter spaces
and neural network settings. This paper develop a novel approach for probing
implicit bias in semi-gradient Q-learning.
