  Planning with a learned model is arguably a key component of intelligence.
There are several challenges in realizing such a component in large-scale
reinforcement learning (RL) problems. One such challenge is dealing effectively
with continuous action spaces when using tree-search planning (e.g., it is not
feasible to consider every action even at just the root node of the tree). In
this paper we present a method for selecting affordances useful for planning --
for learning which small number of actions/options from a continuous space of
actions/options to consider in the tree-expansion process during planning. We
consider affordances that are goal-and-state-conditional mappings to
actions/options as well as unconditional affordances that simply select
actions/options available in all states. Our selection method is gradient
based: we compute gradients through the planning procedure to update the
parameters of the function that represents affordances. Our empirical work
shows that it is feasible to learn to select both primitive-action and option
affordances, and that simultaneously learning to select affordances and
planning with a learned value-equivalent model can outperform model-free RL.
