  We study a Stackelberg game between one attacker and one defender in a
configurable environment. The defender picks a specific environment
configuration. The attacker observes the configuration and attacks via
Reinforcement Learning (RL trained against the observed environment). The
defender's goal is to find the environment with minimum achievable reward for
the attacker. We apply Evolutionary Diversity Optimization (EDO) to generate
diverse population of environments for training. Environments with clearly high
rewards are killed off and replaced by new offsprings to avoid wasting training
time. Diversity not only improves training quality but also fits well with our
RL scenario: RL agents tend to improve gradually, so a slightly worse
environment earlier on may become better later. We demonstrate the
effectiveness of our approach by focusing on a specific application, Active
Directory (AD). AD is the default security management system for Windows domain
networks. AD environment describes an attack graph, where nodes represent
computers/accounts/etc., and edges represent accesses. The attacker aims to
find the best attack path to reach the highest-privilege node. The defender can
change the graph by removing a limited number of edges (revoke accesses). Our
approach generates better defensive plans than the existing approach and scales
better.
