  We study automated intrusion response and formulate the interaction between
an attacker and a defender as an optimal stopping game where attack and defense
strategies evolve through reinforcement learning and self-play. The
game-theoretic modeling enables us to find defender strategies that are
effective against a dynamic attacker, i.e. an attacker that adapts its strategy
in response to the defender strategy. Further, the optimal stopping formulation
allows us to prove that optimal strategies have threshold properties. To obtain
near-optimal defender strategies, we develop Threshold Fictitious Self-Play
(T-FP), a fictitious self-play algorithm that learns Nash equilibria through
stochastic approximation. We show that T-FP outperforms a state-of-the-art
algorithm for our use case. The experimental part of this investigation
includes two systems: a simulation system where defender strategies are
incrementally learned and an emulation system where statistics are collected
that drive simulation runs and where learned strategies are evaluated. We argue
that this approach can produce effective defender strategies for a practical IT
infrastructure.
