  Command and control (C2) paths for issuing commands to malware are sometimes
the only indicators of its existence within networks. Identifying potential C2
channels is often a manually driven process that involves a deep understanding
of cyber tradecraft. Efforts to improve discovery of these channels through
using a reinforcement learning (RL) based approach that learns to automatically
carry out C2 attack campaigns on large networks, where multiple defense layers
are in place serves to drive efficiency for network operators. In this paper,
we model C2 traffic flow as a three-stage process and formulate it as a Markov
decision process (MDP) with the objective to maximize the number of valuable
hosts whose data is exfiltrated. The approach also specifically models payload
and defense mechanisms such as firewalls which is a novel contribution. The
attack paths learned by the RL agent can in turn help the blue team identify
high-priority vulnerabilities and develop improved defense strategies. The
method is evaluated on a large network with more than a thousand hosts and the
results demonstrate that the agent can effectively learn attack paths while
avoiding firewalls.
