  Actor-critic (AC) methods are widely used in reinforcement learning (RL) and
benefit from the flexibility of using any policy gradient method as the actor
and value-based method as the critic. The critic is usually trained by
minimizing the TD error, an objective that is potentially decorrelated with the
true goal of achieving a high reward with the actor. We address this mismatch
by designing a joint objective for training the actor and critic in a
decision-aware fashion. We use the proposed objective to design a generic, AC
algorithm that can easily handle any function approximation. We explicitly
characterize the conditions under which the resulting algorithm guarantees
monotonic policy improvement, regardless of the choice of the policy and critic
parameterization. Instantiating the generic algorithm results in an actor that
involves maximizing a sequence of surrogate functions (similar to TRPO, PPO)
and a critic that involves minimizing a closely connected objective. Using
simple bandit examples, we provably establish the benefit of the proposed
critic objective over the standard squared error. Finally, we empirically
demonstrate the benefit of our decision-aware actor-critic framework on simple
RL problems.
