Abstract: This paper presents a new anomaly detection method based on machine learning. Applicable to host-based intrusion detection systems, this method uses shell commands as audit data. The method employs shell command sequences of different lengths to characterize behavioral patterns of a network user, and constructs multiple sequence libraries to represent the user’s normal behavior profile. In the detection stage, the behavioral patterns in the audit data are mined by a sequence-matching algorithm, and the similarities between the mined patterns and the historical profile are evaluated. These similarities are then smoothed with sliding windows, and the smoothed similarities are used to determine whether the monitored user’s behaviors are normal or anomalous. The results of our experience show the method can achieve higher detection accuracy and shorter detection time than the instance-based method presented by Lane T. The method has been successfully applied in practical host-based intrusion detection systems.

Key words: intrusion detection; machine learning; anomaly detection; shell command