Abstract: Application programming interface (API) is a procedure call interface to operation system resource. API-based behavior features can capture the malicious behaviors of malware variants. However, existing malware detection approaches have a deal of complex operations on constructing and matching. Furthermore, graph matching is adopted in many approaches, which is a nondeterministic polynominal (NP)-complete problem because of computational complexity. To address these problems, a novel approach is proposed to detect malware variants. Firstly, the API of the malware are divided by their functions and parameters. Then, the classified behavior graph (CBG) is constructed from the API call sequences. Finally, the signature based on CBGs for each malware family is generated. Besides, the malware variants are classified by ensemble learning algorithm. Experiments on 1 220 malware samples show that the true positive rate (TPR) is up to 89.0% with the low false positive rate (FPR) 3.7% by ensemble learning.

Key words: