  This paper considers key challenges to using reinforcement learning (RL) with
attack graphs to automate penetration testing in real-world applications from a
systems perspective. RL approaches to automated penetration testing are
actively being developed, but there is no consensus view on the representation
of computer networks with which RL should be interacting. Moreover, there are
significant open challenges to how those representations can be grounded to the
real networks where RL solution methods are applied. This paper elaborates on
representation and grounding using topic challenges of interacting with real
networks in real-time, emulating realistic adversary behavior, and handling
unstable, evolving networks. These challenges are both practical and
mathematical, and they directly concern the reliability and dependability of
penetration testing systems. This paper proposes a layered reference model to
help organize related research and engineering efforts. The presented layered
reference model contrasts traditional models of attack graph workflows because
it is not scoped to a sequential, feed-forward generation and analysis process,
but to broader aspects of lifecycle and continuous deployment. Researchers and
practitioners can use the presented layered reference model as a
first-principles outline to help orient the systems engineering of their
penetration testing systems.
