  Although duality is used extensively in certain fields, such as supervised
learning in machine learning, it has been much less explored in others, such as
reinforcement learning (RL). In this paper, we show how duality is involved in
a variety of RL work, from that which spearheaded the field, such as Richard
Bellman's value iteration, to that which was done within just the past few
years yet has already had significant impact, such as TRPO, A3C, and GAIL. We
show that duality is not uncommon in reinforcement learning, especially when
value iteration, or dynamic programming, is used or when first or second order
approximations are made to transform initially intractable problems into
tractable convex programs.
