  Safety is still one of the major research challenges in reinforcement
learning (RL). In this paper, we address the problem of how to avoid safety
violations of RL agents during exploration in probabilistic and partially
unknown environments. Our approach combines automata learning for Markov
Decision Processes (MDPs) and shield synthesis in an iterative approach.
Initially, the MDP representing the environment is unknown. The agent starts
exploring the environment and collects traces. From the collected traces, we
passively learn MDPs that abstractly represent the safety-relevant aspects of
the environment. Given a learned MDP and a safety specification, we construct a
shield. For each state-action pair within a learned MDP, the shield computes
exact probabilities on how likely it is that executing the action results in
violating the specification from the current state within the next $k$ steps.
After the shield is constructed, the shield is used during runtime and blocks
any actions that induce a too large risk from the agent. The shielded agent
continues to explore the environment and collects new data on the environment.
Iteratively, we use the collected data to learn new MDPs with higher accuracy,
resulting in turn in shields able to prevent more safety violations. We
implemented our approach and present a detailed case study of a Q-learning
agent exploring slippery Gridworlds. In our experiments, we show that as the
agent explores more and more of the environment during training, the improved
learned models lead to shields that are able to prevent many safety violations.
