  In partially observable reinforcement learning, offline training gives access
to latent information which is not available during online training and/or
execution, such as the system state. Asymmetric actor-critic methods exploit
such information by training a history-based policy via a state-based critic.
However, many asymmetric methods lack theoretical foundation, and are only
evaluated on limited domains. We examine the theory of asymmetric actor-critic
methods which use state-based critics, and expose fundamental issues which
undermine the validity of a common variant, and limit its ability to address
partial observability. We propose an unbiased asymmetric actor-critic variant
which is able to exploit state information while remaining theoretically sound,
maintaining the validity of the policy gradient theorem, and introducing no
bias and relatively low variance into the training process. An empirical
evaluation performed on domains which exhibit significant partial observability
confirms our analysis, demonstrating that unbiased asymmetric actor-critic
converges to better policies and/or faster than symmetric and biased asymmetric
baselines.
