Georgia Institute of Technology
Trustworthy Binary Classifications in Dynamic Systems Under Uncertainty
Abstract
dc:description.abstractBinary classifications are commonly used in modeling dynamic systems and are prevalent in machine learning and deep learning applications. These classifications directly inform decision-making, however, uncertainty in modeling can undermine prediction trustworthiness. This dissertation aims to enhance binary classification reliability in dynamic systems under various uncertainties, focusing on safety-critical applications like malware detection and wildfire prediction. In these contexts, predictions inform critical mitigation decisions, necessitating high-fidelity predictions across a range of behaviors that dynamic systems are likely to exhibit. To achieve this, we identify different sources of uncertainty in the modeling pipeline and propose mitigative measures. In malware detection, we perform online uncertainty estimation to monitor predictions post-deployment, recognizing the model's limitations in classifying unknown behavior as benign or malicious. Next, we develop strategies to faithfully capture malware behavior, reducing or managing uncertainty that manifests as an overlap between the positive and negative classes. Finally, shifting the context to a complex dynamic system like wildfire evolution, we address irreducible uncertainty arising from the system's inherent randomness. Here, we propose a novel evaluation criterion that treats uncertainty as a feature rather than a bug. This criterion tests the model's ability to capture macro system behaviors and does not penalize the model if it accurately learns the system's inherent uncertainty. We consolidate our learnings into an uncertainty-based framework which can be used for designing reliable data-driven models that perform binary classifications in uncertain dynamic environments. Our findings apply to alert generation using time series and spatiotemporal data collected from sensors monitoring dynamic system behavior.
Degree
thesis:*- Level thesis:degree_level
- Doctoral
- Department dc:contributor.department
- Electrical and Computer Engineering
- Grantor dc:publisher
- Georgia Institute of Technology
- Year dc:date.issued
- 2024
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Kumar, Harshit
- Advisor dc:contributor.advisor
-
- Mukhopadhyay, Saibal
- Committee members dc:contributor.committeemember
-
- Kim, Hyesoon
- Romberg, Justin
- Hao, Cong (Callie)
- Bracco, Annalisa
Subjects
dc:subject × 3Rights
- Language dc:language.iso
- en_US
Identifiers
dc:identifier.*- Handle dc:identifier.uri
- https://hdl.handle.net/1853/78608