  We consider the problem of learning the optimal threshold policy for control
problems. Threshold policies make control decisions by evaluating whether an
element of the system state exceeds a certain threshold, whose value is
determined by other elements of the system state. By leveraging the monotone
property of threshold policies, we prove that their policy gradients have a
surprisingly simple expression. We use this simple expression to build an
off-policy actor-critic algorithm for learning the optimal threshold policy.
Simulation results show that our policy significantly outperforms other
reinforcement learning algorithms due to its ability to exploit the monotone
property. In addition, we show that the Whittle index, a powerful tool for
restless multi-armed bandit problems, is equivalent to the optimal threshold
policy for an alternative problem. This observation leads to a simple algorithm
that finds the Whittle index by learning the optimal threshold policy in the
alternative problem. Simulation results show that our algorithm learns the
Whittle index much faster than several recent studies that learn the Whittle
index through indirect means.
