  Many applied decision-making problems have a dynamic component: The
policymaker needs not only to choose whom to treat, but also when to start
which treatment. For example, a medical doctor may choose between postponing
treatment (watchful waiting) and prescribing one of several available
treatments during the many visits from a patient. We develop an "advantage
doubly robust" estimator for learning such dynamic treatment rules using
observational data under the assumption of sequential ignorability. We prove
welfare regret bounds that generalize results for doubly robust learning in the
single-step setting, and show promising empirical performance in several
different contexts. Our approach is practical for policy optimization, and does
not need any structural (e.g., Markovian) assumptions.
