Monitoring Buffer Overflow Attacks: A Perennial Task

Monitoring Buffer Overflow Attacks: A Perennial Task

Hossain Shahriar (Queen’s University, Canada) and Mohammad Zulkernine (Queen’s University, Canada)
DOI: 10.4018/978-1-4666-1580-9.ch008


Buffer overflow (BOF) is a well-known, and one of the worst and oldest, vulnerabilities in programs. BOF attacks overwrite data buffers and introduce wide ranges of attacks like execution of arbitrary injected code. Many approaches are applied to mitigate buffer overflow vulnerabilities; however, mitigating BOF vulnerabilities is a perennial task as these vulnerabilities elude the mitigation efforts and appear in the operational programs at run-time. Monitoring is a popular approach for detecting BOF attacks during program execution, and it can prevent or send warnings to take actions for avoiding the consequences of the exploitations. Currently, there is no detailed classification of the proposed monitoring approaches to understand their common characteristics, objectives, and limitations. In this paper, the authors classify runtime BOF attack monitoring and prevention approaches based on seven major characteristics. Finally, these approaches are compared for attack detection coverage based on a set of BOF attack types. The classification will enable researchers and practitioners to select an appropriate BOF monitoring approach or provide guidelines to build a new one.
Chapter Preview


Program Monitor

In general, a program takes inputs, processes them with or without the help of runtime environment (e.g., API calls), and generates outputs as shown in Figure 1(a). A program monitor is deployed in a post-release stage. It provides an additional layer between a program and its execution environment (Figure 1(b)). A monitor passively checks runtime program states for the occurrence of attacks. The underlying assumption is that attack symptoms can be captured by program states. Program states are entities involved in program execution such as program memories containing modified (or unmodified) inputs and outputs, registers, opcode, and attribute of inputs (e.g., sizes). While executing, program states are captured at specific execution points (e.g., beginning of a function call, return from a function) and matched with known states under attacks. Any match (or mismatch) might indicate the occurrence of a successful attack. A monitor might have access to various elements of programs (e.g., source code) and environments (e.g., stack, code, data, inputs, APIs, processors).

Figure 1.

(a) A program without a monitor, (b) program execution after monitoring

Complete Chapter List

Search this Book: