This Master's thesis presents a novel approach for detecting and predicting attack vectors based on genetic programming. The proposed method utilizes a genetic algorithm to evolve a set of rules that predict attack vectors over the system based on caught indicators of compromise. The generated rules are then used to identify potential attack vectors and predict how it started and how it will develop in future. The research aims to improve the accuracy and efficiency of existing methods for attack detection and prediction. The proposed approach is evaluated using real-world attack data and compared against several state-of-the-art techniques. Results indicate that the proposed method outperforms existing approaches in terms of detection accuracy and prediction capability. This research has important implications for the field of cybersecurity and can assist organizations in developing more effective and proactive defense strategies against cyberattacks. Background. Cybersecurity is an increasingly critical issue in today's digital age. Cyberattacks are becoming more sophisticated, making it challenging for traditional defense mechanisms to detect and prevent them. Therefore, it is crucial to develop new and innovative methods for identifying and predicting potential attack vectors. In this context, this Master's thesis presents a novel approach to detecting and predicting attack vectors based on genetic programming. The proposed method aims to improve the accuracy and efficiency of existing approaches to cyberattack detection and prediction. Objectives.This Master’s thesis aims to reach the following objectives: 1. To identify the limitations of existing approaches to cyberattack detection and prevention and propose a novel method based on genetic programming. 2. To develop a genetic programming-based algorithm to evolve a model for attack-vectors prediction. 3. To evaluate the effectiveness of the proposed approach using real-world attack data Methods. The methods used in this Master's thesis combine literature review, data collection, algorithm development, experimentation, data analysis, and recommendations to improving approach to detecting and predicting attack vectors using genetic programming. The research aims to contribute to the field of cybersecurity by advancing our understanding of cyberattack detection and prevention. Results. The proposed method has the potential to enhance the accuracy and efficiency of cyberattack detection and prediction, which can help organizations prevent or mitigate the impact of cyberattacks. Future improvements can include more complex MITRE ATT&CK datasets, including Mobile and ICS matrices. Conclusions. The genetic programming-based algorithm developed in this thesis was shown to be effective in detecting and predicting attack vectors using real-world attack data. The proposed approach has the potential to improve organizations' cybersecurity posture by providing a proactive defense strategy against cyberattacks.

Proactive approaches for preventing attacks through security measurements are crucial for preventing sophisticated attacks. However, proactive measures must employ qualitative security metrics and systemic measurement methodologies to assess security guarantees, as some metrics (e.g., entropy) used for evaluating security guarantees may not capture the capabilities of advanced attackers. Also, many proactive measures (e.g., data pointer protection or data flow integrity) suffer performance bottlenecks. This dissertation identifies and represents attack vectors as metrics using the knowledge from advanced exploits and demonstrates the effectiveness of the metrics by quantifying attack surface and enabling ways to tune performance vs. security of existing defenses by identifying and prioritizing key attack vectors for protection. We measure attack surface by quantifying the impact of fine-grained Address Space Layout Randomization (ASLR) on code reuse attacks under the Just-In-Time Return-Oriented Programming (JITROP) threat model. We conduct a comprehensive measurement study with five fine-grained ASLR tools, 20 applications including six browsers, one browser engine, and 25 dynamic libraries. Experiments show that attackers only need several seconds (1.5-3.5) to find various code reuse gadgets such as the Turing Complete gadget set. Experiments also suggest that some code pointer leaks allow attackers to find gadgets more quickly than others. Besides, the instruction-level single-round randomization can restrict Turing Complete operations by preventing up to 90% of gadgets. This dissertation also identifies and prioritizes critical data pointers for protection to enable the capability to tune between performance vs. security. We apply seven rule-based heuristics to prioritize externally manipulatable sensitive data objects/pointers. Our evaluations using 33 ground truths vulnerable data objects/pointers show the successful detection of 32 ground truths with a 42% performance overhead reduction compared to AddressSanitizer. Our results also suggest that sensitive data objects are as low as 3%, and on average, 82% of data objects do not need protection for real-world applications. / Doctor of Philosophy / Proactive approaches for preventing attacks through security measurements are crucial to prevent advanced attacks because reactive measures can become challenging, especially when attackers enter sophisticated attack phases. A key challenge for the proactive measures is the identification of representative metrics and measurement methodologies to assess security guarantees, as some metrics used for evaluating security guarantees may not capture the capabilities of advanced attackers. Also, many proactive measures suffer performance bottlenecks. This dissertation identifies and represents attack elements as metrics using the knowledge from advanced exploits and demonstrates the effectiveness of the metrics by quantifying attack surface and enabling the capability to tune performance vs. security of existing defenses by identifying and prioritizing key attack elements. We measure the attack surface of various software applications by quantifying the available attack elements of code reuse attacks in the presence of fine-grained Address Space Layout Randomization (ASLR), a defense in modern operating systems. ASLR makes code reuse attacks difficult by making the attack components unavailable. We perform a comprehensive measurement study with five fine-grained ASLR tools, real-world applications, and libraries under an influential code reuse attack model. Experiments show that attackers only need several seconds (1.5-3.5) to find various code reuse elements. Results also show the influence of one attack element over another and one defense strategy over another strategy. This dissertation also applies seven rule-based heuristics to prioritize externally manipulatable sensitive data objects/pointers – a type of attack element – to enable the capability to tune between performance vs. security. Our evaluations using 33 ground truths vulnerable data objects/pointers show the successful identification of 32 ground truths with a 42% performance overhead reduction compared to AddressSanitizer, a memory error detector. Our results also suggest that sensitive data objects are as low as 3% of all objects, and on average, 82% of objects do not need protection for real-world applications.