Article Preview
Top1. Introduction
With the advancement of cutting-edge technologies, our lives are getting easier but the same technology can be exploited in a wrong way that can cause immense harm to an organization or an individual. Thus, anomaly detection becomes an essential task to make our systems and networks secure. At the same time, it is also crucial to secure other rare events from any kind of exploitation. These rare events may have great significance but can be extremely hard to find, which may be the result of increasing vulnerabilities due to the emergence of complex technologies. In modern systems log files hold the system state and significant information on various execution paths.
Though the data in log files might seem to be homogeneous, it may contain some unusual anomaly that might not be visible to the user. To address this issue, methods for anomaly detection come into play. Anomaly detection helps to identify any anomalous behavior in our data or unexpected pattern that should not conform to an expected pattern. System Logs can play a critical part in this quest for anomaly detection. System logs record all the information on an active running process. So, if any failure occurs or any anomaly happens, it gets recorded in the system logs.
In an event of normal functioning, the log files contain homogeneous data, but any unusual behaviour or an anomaly results in a pattern change in the log files. Therefore, we can harness this property into a method to detect anomalies by looking at the system logs for any unusual pattern.
But, log files are created in huge amounts in a system, and parsing them manually in search of any anomaly might not be possible for a human expert. Here, we need an automated process that can be used for improved efficiency. Anomaly detection addresses unpredictable or uncertain, rare and minor events. This increases the complexity of the problem for the detection methods. The rarity and the heterogeneous nature of the anomalies makes it difficult to identify and leads to false classification of normal events as anomalies. Suppose you are working as a system admin at an e-commerce giant. There can be an issue in the front-end that stops your customer from buying things in your platform. How do you know if your customers spending suddenly drops when your services still run perfectly normal? That’s when anomaly detection comes in. Although a large number of methods have been introduced over the years (Breunig et al., 2000; Liu et al., 2012), reducing the false positives and increasing the recall rates for detecting anomalies are an important yet difficult challenge to address. Since the data generated by logs are of higher dimensions and anomaly detection in higher dimensions has been a long standing problem (Zimek et al., 2012), performing the detection in a subspace of original feature (Keller et al., 2012; Lazarevic & Kumar, 2005; Liu et al., 2012) or constructed features seems like a straightforward solution. However, identifying higher order, heterogeneous and non-linear feature interaction and coupling remains a major challenge for anomaly detection. Furthermore, the previously known methods of subspace based methods and feature selection based methods (Altalhi & Gutub, 2021; Pang et al., 2018; Pang et al., 2017) does not preserve the proper information. Thus it may become challenging for anomaly detection due to the heterogeneity of the anomalies. These previously known methods also do not address the challenges of detecting anomalies that have spatial, temporal or graph-based interdependent relationships among them. The rarity of the anomalies makes the dataset of the anomalies susceptible to noisy instances. The main challenge is to significantly identify the noises as they can be distributed in the data space irregularly. This challenge poses a major obstacle for traditional machine learning techniques(Juvonen et al., 2015). These challenges posed by anomaly detection in traditional methods can be addressed by the Convolutional Neural Networks (CNN).
Previous approaches used Long Short Term Memory (LSTM), Principal Component Analysis (PCA) based anomaly detection, invariant mining, one-class Support Vector Machine (SVM), isolation forests to detect anomaly in the system log files (Du et al., 2017; Juvonen et al., 2015; Liang et al., 2007).