Web application attack volume, complexity, and costs have risen as people, companies, and entire industries move online. Solutions implemented to defend web applications against malicious activity have traditionally been implemented at the network or host layer. While this is helpful for detecting some attacks, it does not provide the gran- ularity to see malicious behavior occurring at the application layer. The AppSensor project, an application level intrusion detection system (IDS), is an example of a tool that operates in this layer. AppSensor monitors users within the application by observing activity in suspicious areas not able to be seen by traditional network layer tools. This thesis aims to improve the state of web application security by supporting the development of the AppSensor project. Specifically, this thesis entails contributing a rules-based analysis engine to provide a new method for determining whether suspicious activity constitutes an attack. The rules-based method aggregates information from multiple sources into a logical rule to identify malicious activity, as opposed to relying on a single source of information. The rules-based analysis engine is designed to offer more flexible configuration for administrators and more accurate results than the incumbent analysis engine. Tests indicate that the new engine should not hamper the performance of AppSensor and use cases highlight how rules can be leveraged for more accurate results.

The rampant growth of the Internet has been coupled with an equivalent growth in cyber crime over the Internet. With our increased reliance on the Internet for commerce, social networking, information acquisition, and information exchange, intruders have found financial, political, and military motives for their actions. Network Intrusion Detection Systems (NIDSs) intercept the traffic at an organization’s periphery and try to detect intrusion attempts. Signature-based NIDSs compare the packet to a signature database consisting of known attacks and malicious packet fingerprints. The signatures use regular expressions to model these intrusion activities. This thesis presents a memory efficient pattern matching system for the class of regular expressions appearing frequently in the NIDS signatures. Proposed Cascaded Automata Architecture is based on two stage automata. The first stage recognizes the sub-strings and character classes present in the regular expression. The second stage consumes symbol generated by the first stage upon receiving input traffic symbols. The basic idea is to utilize the research done on string matching problem for regular expression pattern matching. We formally model the class of regular expressions mostly found in NIDS signatures. The challenges involved in using string matching algorithms for regular expression matching has been presented. We introduce length-bound transitions, counter-based states, and associated counter arrays in the second stage automata to address these challenges. The system uses length information along with counter arrays to keep track of overlapped sub-strings and character class based transition. We present efficient implementation techniques for counter arrays. The evaluation of the architecture on practical expressions from Snort rule set showed compression in number of states between 50% to 85%. Because of its smaller memory footprint, our solution is suitable for both software based implementations on network chips as well as FPGA based designs.