  Controller synthesis is in essence a case of model-based planning for
non-deterministic environments in which plans (actually ''strategies'') are
meant to preserve system goals indefinitely. In the case of supervisory control
environments are specified as the parallel composition of state machines and
valid strategies are required to be ''non-blocking'' (i.e., always enabling the
environment to reach certain marked states) in addition to safe (i.e., keep the
system within a safe zone). Recently, On-the-fly Directed Controller Synthesis
techniques were proposed to avoid the exploration of the entire -and
exponentially large-environment space, at the cost of non-maximal
permissiveness, to either find a strategy or conclude that there is none. The
incremental exploration of the plant is currently guided by a
domain-independent human-designed heuristic. In this work, we propose a new
method for obtaining heuristics based on Reinforcement Learning (RL). The
synthesis algorithm is thus framed as an RL task with an unbounded action space
and a modified version of DQN is used. With a simple and general set of
features that abstracts both states and actions, we show that it is possible to
learn heuristics on small versions of a problem that generalize to the larger
instances, effectively doing zero-shot policy transfer. Our agents learn from
scratch in a highly partially observable RL task and outperform the existing
heuristic overall, in instances unseen during training.
