  Penetration testing the organised attack of a computer system in order to
test existing defences has been used extensively to evaluate network security.
This is a time consuming process and requires in-depth knowledge for the
establishment of a strategy that resembles a real cyber-attack. This paper
presents a novel deep reinforcement learning architecture with hierarchically
structured agents called HA-DRL, which employs an algebraic action
decomposition strategy to address the large discrete action space of an
autonomous penetration testing simulator where the number of actions is
exponentially increased with the complexity of the designed cybersecurity
network. The proposed architecture is shown to find the optimal attacking
policy faster and more stably than a conventional deep Q-learning agent which
is commonly used as a method to apply artificial intelligence in automatic
penetration testing.
