  Learning With Opponent-Learning Awareness (LOLA) (Foerster et al. [2018a]) is
a multi-agent reinforcement learning algorithm that typically learns
reciprocity-based cooperation in partially competitive environments. However,
LOLA often fails to learn such behaviour on more complex policy spaces
parameterized by neural networks, partly because the update rule is sensitive
to the policy parameterization. This problem is especially pronounced in the
opponent modeling setting, where the opponent's policy is unknown and must be
inferred from observations; in such settings, LOLA is ill-specified because
behaviorally equivalent opponent policies can result in non-equivalent updates.
To address this shortcoming, we reinterpret LOLA as approximating a proximal
operator, and then derive a new algorithm, proximal LOLA (POLA), which uses the
proximal formulation directly. Unlike LOLA, the POLA updates are
parameterization invariant, in the sense that when the proximal objective has a
unique optimum, behaviorally equivalent policies result in behaviorally
equivalent updates. We then present practical approximations to the ideal POLA
update, which we evaluate in several partially competitive environments with
function approximation and opponent modeling. This empirically demonstrates
that POLA achieves reciprocity-based cooperation more reliably than LOLA.
