  Software vulnerabilities are a challenge in cybersecurity. Manual security
patches are often difficult and slow to be deployed, while new vulnerabilities
are created. Binary code vulnerability detection is less studied and more
complex compared to source code, and this has important practical implications.
Deep learning has become an efficient and powerful tool in the security domain,
where it provides end-to-end and accurate prediction. Modern deep learning
approaches learn the program semantics through sequence and graph neural
networks, using various intermediate representation of programs, such as
abstract syntax trees (AST) or control flow graphs (CFG). Due to the complex
nature of program execution, the output of an execution depends on the many
program states and inputs. Also, a CFG generated from static analysis can be an
overestimation of the true program flow. Moreover, the size of programs often
does not allow a graph neural network with fixed layers to aggregate global
information. To address these issues, we propose DeepEXE, an agent-based
implicit neural network that mimics the execution path of a program. We use
reinforcement learning to enhance the branching decision at every program state
transition and create a dynamic environment to learn the dependency between a
vulnerability and certain program states. An implicitly defined neural network
enables nearly infinite state transitions until convergence, which captures the
structural information at a higher level. The experiments are conducted on two
semi-synthetic and two real-world datasets. We show that DeepEXE is an accurate
and efficient method and outperforms the state-of-the-art vulnerability
detection methods.
