  We propose a novel approach to the problem of controller design for
environments modeled as Markov decision processes (MDPs). Specifically, we
consider a hierarchical MDP a graph with each vertex populated by an MDP called
a "room". We first apply deep reinforcement learning (DRL) to obtain low-level
policies for each room, scaling to large rooms of unknown structure. We then
apply reactive synthesis to obtain a high-level planner that chooses which
low-level policy to execute in each room. The central challenge in synthesizing
the planner is the need for modeling rooms. We address this challenge by
developing a DRL procedure to train concise "latent" policies together with PAC
guarantees on their performance. Unlike previous approaches, ours circumvents a
model distillation step. Our approach combats sparse rewards in DRL and enables
reusability of low-level policies. We demonstrate feasibility in a case study
involving agent navigation amid moving obstacles.
