  Soft Actor-Critic (SAC) is considered the state-of-the-art algorithm in
continuous action space settings. It uses the maximum entropy framework for
efficiency and stability, and applies a heuristic temperature Lagrange term to
tune the temperature $\alpha$, which determines how "soft" the policy should
be. It is counter-intuitive that empirical evidence shows SAC does not perform
well in discrete domains. In this paper we investigate the possible
explanations for this phenomenon and propose Target Entropy Scheduled SAC
(TES-SAC), an annealing method for the target entropy parameter applied on SAC.
Target entropy is a constant in the temperature Lagrange term and represents
the target policy entropy in discrete SAC. We compare our method on Atari 2600
games with different constant target entropy SAC, and analyze on how our
scheduling affects SAC.
