  Recently, with the booming development of software industry, more and more
malware variants are designed to perform malicious behaviors. The evolution of
malware makes it difficult to detect using traditional signature-based methods.
Moreover, malware detection has important effect on system security. In this
paper, we present SCGDet, which is a novel malware detection method based on
system call graph model (SCGM). We first develop a system call pruning method,
which can exclude system calls that have little impact on malware detection.
Then we propose the SCGM, which can capture the semantic features of run-time
program by grouping the system calls based on the reachability relation. We aim
to obtain the generic representation of malicious behaviors with similar system
call patterns. We evaluate the performance of SCGDet using different machine
learning algorithms on the dataset including 854 malware samples and 740 benign
samples. Compared with the traditional n-gram method, the SCGDet has the
smaller feature space, the higher detection accuracy and the lower false
positives. Experimental results show that SCGDet can reduce the average FPR of
14.75% and improve the average Accuracy of 8.887%, and can obtain a TPR of
97.44%, an FPR of 1.96% and an Accuracy of 97.78% in the best case.
