  Most model-free reinforcement learning methods leverage state representations
(embeddings) for generalization, but either ignore structure in the space of
actions or assume the structure is provided a priori. We show how a policy can
be decomposed into a component that acts in a low-dimensional space of action
representations and a component that transforms these representations into
actual actions. These representations improve generalization over large, finite
action sets by allowing the agent to infer the outcomes of actions similar to
actions already taken. We provide an algorithm to both learn and use action
representations and provide conditions for its convergence. The efficacy of the
proposed method is demonstrated on large-scale real-world problems.
