Intrusion detection is the practice of examining information from computers and networks to identify cyberattacks. It is an important topic in practice, since the frequency and consequences of cyberattacks continues to increase and affect organizations. It is important for research, since many problems exist for intrusion detection systems. Intrusion detection systems monitor large volumes of data and frequently generate false positives. This results in additional effort for security analysts to review and interpret alerts. After long hours spent reviewing alerts, security analysts become fatigued and make bad decisions. There is currently no approach to intrusion detection that reduces the workload of human analysts by providing a probabilistic prediction that a computer is experiencing a cyberattack. This research addressed this problem by estimating the probability that a computer system was being attacked, rather than alerting on individual events. This research combined concepts from cyber situation awareness by applying clustering ensembles, probability analysis, and active learning. The unique contribution of this research is that it provides a higher level of meaning for intrusion alerts than traditional approaches. Three experiments were conducted in the course of this research to demonstrate the feasibility of these concepts. The first experiment evaluated cluster generation approaches that provided multiple perspectives of network events using unsupervised machine learning. The second experiment developed and evaluated a method for detecting anomalies from the clustering results. This experiment also determined the probability that a computer system was being attacked. Finally, the third experiment integrated active learning into the anomaly detection results and evaluated its effectiveness in improving the accuracy. This research demonstrated that clustering ensembles with probabilistic analysis were effective for identifying normal events. Abnormal events remained uncertain and were assigned a belief. By aggregating the belief to find the probability that a computer system was under attack, the resulting probability was highly accurate for the source IP addresses and reasonably accurate for the destination IP addresses. Active learning, which simulated feedback from a human analyst, eliminated the residual error for the destination IP addresses with a low number of events that required labeling.

<div>Cyber-based infrastructures and systems serve as the operational backbone of many industries and resilience of such systems against cyber-attacks is of paramount importance. As the complexity and scale of the Cyber-based Systems (CBSs) has increased many folds over the years, the attack surface has also been widened, making CBSs more vulnerable to cyber-attacks. This dissertation addresses the challenges in post intrusion security management operations of threat detection and threat response in the networks connecting CBSs. In threat detection, the increase in scale of cyber networks and the rise in sophistication of cyber-attacks has introduced several challenges. The primary challenge is the requirement to detect complex multi-stage cyber-attacks in realtime by processing the immense amount of traffic produced by present-day networks. In threat response, the issue of delay in responding to cyber-attacks and the functional interdependencies among different systems of CBS has been observed to have catastrophic effects, as a cyber attack that compromises one constituent system of a CBS can quickly disseminate to others. This can result in a cascade effect that can impair the operability of the entire CBS. To address the challenges in threat detection, this dissertation proposes PRISM, a hierarchical threat detection architecture that uses a novel attacker behavior model-based sampling technique to minimize the realtime traffic processing overhead. PRISM has a unique multi-layered architecture that monitors network traffic distributedly to provide efficiency in processing and modularity in design. PRISM employs a Hidden Markov Model-based prediction mechanism to identify multi-stage attacks and ascertain the attack progression for a proactive response. Furthermore, PRISM introduces a stream management procedure that rectifies the issue of alert reordering when collected from distributed alert reporting systems. To address the challenges in threat response, this dissertation presents TRAP, a novel threat response and recovery architecture that localizes the cyber-attack in a timely manner, and simultaneously recovers the affected system functionality. The dissertation presents comprehensive performance evaluation of PRISM and TRAP through extensive experimentation, and shows their effectiveness in identifying threats and responding to them while achieving all of their design objectives.</div>