  Current frameworks for training offensive penetration testing agents with
deep reinforcement learning struggle to produce agents that perform well in
real-world scenarios, due to the reality gap in simulation-based frameworks and
the lack of scalability in emulation-based frameworks. Additionally, existing
frameworks often use an unrealistic metric that measures the agents'
performance on the training data. NASimEmu, a new framework introduced in this
paper, addresses these issues by providing both a simulator and an emulator
with a shared interface. This approach allows agents to be trained in
simulation and deployed in the emulator, thus verifying the realism of the used
abstraction. Our framework promotes the development of general agents that can
transfer to novel scenarios unseen during their training. For the simulation
part, we adopt an existing simulator NASim and enhance its realism. The
emulator is implemented with industry-level tools, such as Vagrant, VirtualBox,
and Metasploit. Experiments demonstrate that a simulation-trained agent can be
deployed in emulation, and we show how to use the framework to train a general
agent that transfers into novel, structurally different scenarios. NASimEmu is
available as open-source.
