  We propose a deep learning approach for identifying malware families using
the function call graphs of x86 assembly instructions. Though prior work on
static call graph analysis exists, very little involves the application of
modern, principled feature learning techniques to the problem. In this paper,
we introduce a system utilizing an executable's function call graph where
function representations are obtained by way of a recurrent neural network
(RNN) autoencoder which maps sequences of x86 instructions into dense, latent
vectors. These function embeddings are then modeled as vertices in a graph with
edges indicating call dependencies. Capturing rich, node-level representations
as well as global, topological properties of an executable file greatly
improves malware family detection rates and contributes to a more principled
approach to the problem in a way that deliberately avoids tedious feature
engineering and domain expertise. We test our approach by performing several
experiments on a Microsoft malware classification data set and achieve
excellent separation between malware families with a classification accuracy of
99.41%.
