  We study online adaptive policy selection in systems with time-varying costs
and dynamics. We develop the Gradient-based Adaptive Policy Selection (GAPS)
algorithm together with a general analytical framework for online policy
selection via online optimization. Under our proposed notion of contractive
policy classes, we show that GAPS approximates the behavior of an ideal online
gradient descent algorithm on the policy parameters while requiring less
information and computation. When convexity holds, our algorithm is the first
to achieve optimal policy regret. When convexity does not hold, we provide the
first local regret bound for online policy selection. Our numerical experiments
show that GAPS can adapt to changing environments more quickly than existing
benchmarks.
