  There has been considerable and growing interest in applying machine learning
for cyber defenses. One promising approach has been to apply natural language
processing techniques to analyze logs data for suspicious behavior. A natural
question arises to how robust these systems are to adversarial attacks. Defense
against sophisticated attack is of particular concern for cyber defenses. In
this paper, we develop a testing framework to evaluate adversarial robustness
of machine learning cyber defenses, particularly those focused on log data. Our
framework uses techniques from deep reinforcement learning and adversarial
natural language processing. We validate our framework using a publicly
available dataset and demonstrate that our adversarial attack does succeed
against the target systems, revealing a potential vulnerability. We apply our
framework to analyze the influence of different levels of dropout
regularization and find that higher dropout levels increases robustness.
Moreover 90% dropout probability exhibited the highest level of robustness by a
significant margin, which suggests unusually high dropout may be necessary to
properly protect against adversarial attacks.
