Advanced Threat Detection Based on Big Data Technologies

Background

At the recent World Economic Forum (WEF) 2016, the growing number of cyber attacks was a major topic of concern. According to its 11th annual global risks report, cyber-attacks are ranked in the list of top ten threats in 140 economies (“The Global Risks” 2016). Failure in addressing and understanding these cyber attacks could affect economic sectors, national economies and global enterprises. Most of the firewall and other network-based security products provide mature and robust logging capabilities. Since the perimeter security is not enough, most of the security programs start with analyzing logs from the devices at the edge of the network. Nowadays most of the hackers of cyber conflicts are well organized with specific objectives, goals and having strong teams that are heavily funded. They are targeting information and communication systems of industrial, government, military and other private organizations. Also they are willing to use any amount of money, time to become expertise to reach their goals.

So understanding the limitations and problems of current technologies facing against advanced persistent threats (APTs) is important. APTs are significantly different from traditional attacks due to their own characteristics (Virvilis et al, 2014).

• •

  APTs can bypass the majority of network intrusion detection systems and signature-based end points because they are using zero-day.

• •

  The time taken by these attacks is outside the limited window of time of these detection systems due to the fact that they are generally spread over a wide period of time.

• •

  Attackers are willing to spend significant time on focusing a particular target and explore all possible attack paths until they manage to overcome its defence.

• •

  Attacks are highly selective. Targeted victims are selected very carefully, usually departments of an organization which are less likely to identify and report an attack and are nontechnical.

• •

  Based on the analysis of the major APT attacks, it is observed that they are well-supported by nation-states that have significant capabilities enabled (covert physical access, manufacturing, intelligence collection) for cyber-attacks.

Due to these characteristics, present solutions of cyber security will fail to provide an effective defence against such attacks. Signature-based approach is used most widely used in intrusion detection. It is a simple testing methodology using known attack patterns where detection is based on small variations of attack patterns. But it has substantial limitations in intrusion detection systems against advanced persistent threats.