  The number of malicious software (malware) is growing out of control.
Syntactic signature based detection cannot cope with such growth and manual
construction of malware signature databases needs to be replaced by computer
learning based approaches. Currently, a single modern signature capturing the
semantics of a malicious behavior can be used to replace an arbitrarily large
number of old-fashioned syntactical signatures. However teaching computers to
learn such behaviors is a challenge. Existing work relies on dynamic analysis
to extract malicious behaviors, but such technique does not guarantee the
coverage of all behaviors. To sidestep this limitation we show how to learn
malware signatures using static reachability analysis. The idea is to model
binary programs using pushdown systems (that can be used to model the stack
operations occurring during the binary code execution), use reachability
analysis to extract behaviors in the form of trees, and use subtrees that are
common among the trees extracted from a training set of malware files as
signatures. To detect malware we propose to use a tree automaton to compactly
store malicious behavior trees and check if any of the subtrees extracted from
the file under analysis is malicious. Experimental data shows that our approach
can be used to learn signatures from a training set of malware files and use
them to detect a test set of malware that is 5 times the size of the training
set.
