Testing is an indispensable mechanism for assuring software quality. One of the key issues in testing is to obtain a test data set that is able to effectively test an implementation. An adequate test data set consists of test cases that can expose faults in a software implementation. Mutation-based testing can be employed to obtain adequate test data sets, and numerous mutation operators have been proposed to date to measure the adequacy of test data sets that reveal functional faults. However, implementations that pass functionality tests are still vulnerable to malicious attacks. Despite the rigorous use of various existing testing techniques, many vulnerabilities are discovered after the deployment of software implementations, such as buffer overflows (BOF), SQL injections, and format string bugs (FSB). Successful exploitations of these vulnerabilities may result in severe consequences such as denial of services, application state corruptions, and information leakage. Many approaches have been proposed to detect these vulnerabilities. Unfortunately, very few approaches address the issue of testing implementations against vulnerabilities. Moreover, these approaches do not provide an indication whether a test data set is adequate for vulnerability testing or not. We believe that bringing the idea of traditional functional test adequacy to vulnerability testing can help address the issue of test adequacy. In this thesis, we apply the idea of mutation-based adequate testing to perform vulnerability testing of buffer overflows, SQL injections, and format string bugs. We propose mutation operators to force the generation of adequate test data sets for these vulnerabilities. The operators mutate source code to inject the vulnerabilities in the library function calls and unsafe implementation language elements. The mutants generated by the operators are killed by test cases that expose these vulnerabilities. We propose distinguishing or killing criteria for mutants that consider varying symptoms of exploitations. Three prototype tools are developed to automatically generate mutants and perform mutation analysis with input test cases and the effectiveness of the proposed operators is evaluated on several open source programs containing known vulnerabilities. The results indicate that the proposed operators are effective for testing the vulnerabilities, and the mutation-based vulnerability testing process ensures the quality of the applications against these vulnerabilities. / Thesis (Master, Computing) -- Queen's University, 2008-08-18 13:53:04.036