Date of Award

5-2002

Document Type

Thesis

Degree Name

Master of Science (MS)

Department

Computer Engineering and Sciences

First Advisor

James A. Whittaker

Second Advisor

Alan A. Jorgensen

Third Advisor

Fredric M. Ham

Fourth Advisor

William D. Shoaff

Abstract

Buffer overflows have been the most common form of security vulnerability for the last ten years. More over, buffer overflow vulnerabilities enable the type of exploits that dominate remote network penetration. As our reliance on commercial third party software is critical in the current computing environment one must consider the question of how these vulnerabilities are discovered in released proprietary software. This thesis presents research focused on the fundamental issues surrounding the buffer overflow vulnerability. The objective is to analyze and understand the technical nature of this type of vulnerability and, on the basis of this, develop an efficient generic method that can improve the detection of this software flaw in released, proprietary software systems. The work is performed from the perspective of a security auditor searching for a single vulnerability in a released program, a different approach compared to the many previous studies that focus on both static source code analysis and run time fault injection. First, for systems that include commercial off-the-shelf software components, we perform a systematic review of buffer overflow exploit data and develop a classification hierarchy. The goal of this new taxonomy is to provide a tool to assist the auditor in developing the heuristic elements for exploratory testing. Second, we propose that a signature analysis of a disassembled binary executable can lead to the discovery of a buffer overflow vulnerability. In support of this argument we demonstrate a methodology that can be used on closed source proprietary software where only the executable binary image is available. In this case, the key selling point is not the potential rapid automated detection of a buffer overflow vulnerability but the proof of concept that security flaws can be detected by binary scanning techniques.

Comments

Copyright held by author

Download

Share

COinS