  We show that the simplest actor-critic method -- a linear softmax policy
updated with TD through interaction with a linear MDP, but featuring no
explicit regularization or exploration -- does not merely find an optimal
policy, but moreover prefers high entropy optimal policies. To demonstrate the
strength of this bias, the algorithm not only has no regularization, no
projections, and no exploration like $\epsilon$-greedy, but is moreover trained
on a single trajectory with no resets. The key consequence of the high entropy
bias is that uniform mixing assumptions on the MDP, which exist in some form in
all prior work, can be dropped: the implicit regularization of the high entropy
bias is enough to ensure that all chains mix and an optimal policy is reached
with high probability. As auxiliary contributions, this work decouples concerns
between the actor and critic by writing the actor update as an explicit mirror
descent, provides tools to uniformly bound mixing times within KL balls of
policy space, and provides a projection-free TD analysis with its own implicit
bias which can be run from an unmixed starting distribution.
