  In this study, we investigate the limits of the current state of the art AI
system for detecting buffer overflows and compare it with current static
analysis tools. To do so, we developed a code generator, s-bAbI, capable of
producing an arbitrarily large number of code samples of controlled complexity.
We found that the static analysis engines we examined have good precision, but
poor recall on this dataset, except for a sound static analyzer that has good
precision and recall. We found that the state of the art AI system, a memory
network modeled after Choi et al. [1], can achieve similar performance to the
static analysis engines, but requires an exhaustive amount of training data in
order to do so. Our work points towards future approaches that may solve these
problems; namely, using representations of code that can capture appropriate
scope information and using deep learning methods that are able to perform
arithmetic operations.
