  Malware analysis has been extensively investigated as the number and types of
malware has increased dramatically. However, most previous studies use
end-to-end systems to detect whether a sample is malicious, or to identify its
malware family. In this paper, we propose a neural network framework composed
of an embedder, an encoder, and a filter to learn malware representations from
characteristic execution sequences for malware family classification. The
embedder uses BERT and Sent2Vec, state-of-the-art embedding modules, to capture
relations within a single API call and among consecutive API calls in an
execution trace. The encoder comprises gated recurrent units (GRU) to preserve
the ordinal position of API calls and a self-attention mechanism for comparing
intra-relations among different positions of API calls. The filter identifies
representative API calls to build the malware representation. We conduct broad
experiments to determine the influence of individual framework components. The
results show that the proposed framework outperforms the baselines, and also
demonstrates that considering Sent2Vec to learn complete API call embeddings
and GRU to explicitly preserve ordinal information yields more information and
thus significant improvements. Also, the proposed approach effectively
classifies new malicious execution traces on the basis of similarities with
previously collected families.
