Monitoring Buffer Overflow Attacks: A Perennial Task

Introduction

A vulnerable program can be exploited at runtime by providing specially crafted inputs. Buffer overflow (BOF) is a well known and one of the worst and oldest vulnerabilities in programs (Aleph One, 1996). It allows attackers to overflow data buffers that might be exploited to execute arbitrary code. Several mitigation techniques are widely used to mitigate BOF vulnerabilities. These include static analysis (e.g., Hackett et al., 2006), testing (e.g., Xu et al., 2008), and fixing of vulnerable code (e.g., Dahn et al., 2003). However, BOF vulnerabilities are widely discovered in programs (e.g., CVE, 2010). Moreover, some BOF vulnerability exploitations (or attacks) might not appear until a program is operational. Thus, BOF attack detection is a perennial task.

Monitoring is a widely used technique that can detect BOF attacks at an early stage and mitigate some of the consequences at runtime. In a monitoring approach, vulnerability exploitation symptoms are checked by comparing the current state of a program with a known state under attack. When there is a match (or mismatch) between the two states, a successful exploitation of a particular vulnerability occurs. A program might be stopped for further execution. A monitor remains silent as long as a program is not under an attack at the cost of additional memories and execution time (e.g., Jones et al., 1997). Nevertheless, a program monitor is accurate in detecting attacks compared to other complementary mitigation techniques such as static analysis. This unique feature makes it a useful prevention mechanism in a deployed program.

Although many monitoring approaches have been introduced in the literature to detect the exploitations of BOF vulnerabilities (or attacks) (e.g., Berger et al., 2006; Chiueh et al., 2001), there is no classification to understand the common characteristics, objectives, and limitations of these approaches. Moreover, the lack of a comprehensive comparative study provides little or no direction on choosing the appropriate monitoring techniques for particular needs.

In this paper, we perform an extensive survey on the state of the art runtime monitoring approaches that detect BOF attacks¹. We classify the monitoring approaches based on seven most common characteristics: monitoring objective, program state utilization, implementation mechanism, environmental change, attack response, monitor security, and overhead. Moreover, for each of the characteristics, we further classify the current work to identify fine grained features that might be present in BOF monitoring techniques. We then perform a comparative analysis of existing approaches for BOF attack detection coverage. We identify BOF attack types based on both vulnerable program code (operation, data type, overflow among object members, and pointer arithmetic) and runtime state (BOF location, BOF magnitude). The survey will help secure software developers, researchers, and practitioners to select a tool from the existing monitoring approaches by highlighting the BOF attack type detection capabilities. Moreover, it will provide a guideline to build a new monitoring technique based on their particular application needs.

This paper is organized as follows: the next section provides an overview of program monitor and BOF attack. Then we discuss the classification of the monitoring works followed by comparison of the works based on BOF attack types. We then review other similar efforts on comparing BOF attack monitoring approaches. Finally, we draw conclusions.