Back to search

University of Southampton

Advances in hazard management for machine learning based safety-critical systems operating in unpredictable environments

Abstract

dc:description.abstract

Autonomous systems deployed in open, unpredictable environments face a class of engineering challenges that conventional safety analysis frameworks are structurally ill-equipped to address. The problem is that the complexity of combining highly complicated operational environment and a non-deterministic system of interest (SoI) do not organise their failure modes according to any prescribed safety method's epistemology that is inherently derived from studying and engineering deterministic, closed systems' behaviours. The SoI and its environment will produce interactions that fall entirely outside the deterministic-systems-based method's epistemological assumptions, and those interactions may simply never be thought of. So, the coverage of the analysis is bounded by the analyst's attention distribution (what the method directs them to look at) rather than by the interaction space of the environment (what the environment can actually produce). <br/><br/>Existing methods, HAZOP, STAMP/STPA, and FRAM, operate under implicit epistemological and ontological assumptions of control-bounded, optimised scoping failure spaces, producing hazard analyses whose coverage is constrained by the solutioning fixation distribution of the analyst rather than by the actual complexity of the operational environment. This thesis identifies and addresses three interrelated gaps: the Black Swan Prediction Gap, arising from the absence of systematic techniques for proactive discovery and impact assessment of novel, low-salience interactions; the Dataset Design Gap, arising from the lack of safety-driven, standardised approaches to ensure demonstrable Black Swan coverage in machine learning (ML) training, validation, and test datasets; and the Semantic Gap, arising from the disconnect between textual safety requirements and large-scale pictorial datasets, which limits auditable traceability and dataset trustworthiness. <br/><br/>To address these gaps, the thesis proposes the AIC Systems Approach: a six-stage, systems-theoretic safety engineering framework underpinned by a candidate Organisational Universe Meta Model ontology, which conceptualises engineered reality as rule-governed CONTENT organised through RULES set, Complexes, Systems, Organisational Experience (OrgXP), organisational friction, and predictability/order. Building on this ontology, the approach operationalises AIC (Appreciation, Influence, Control) systems theory through Systems-Theoretic Chain-of-Thought (STCoT) and produces AIC Systems Approach, a structured, general-systems-model-grounded predictive reasoning protocol for identifying typical, atypical, and Black Swan candidate scenarios in non-deterministic SoI operating in unpredictable environment.<br/><br/>The thesis introduces the concept of Architect’s Analytical Attention Distribution (ATD): a heuristic mental model of inquired real-world OrgXP Uncertainty Distribution Profile (UDP) that examines the frequency distribution of predicted behaviours and hazards across an analysis method's ontological categories. Three ATD regimes are defined; Normal, Uniform, and Long-tailed, and associated with Vertical, Lateral, and T-thinking modes respectively. ATD shape is quantified through statistical properties including excess kurtosis, coverage blindness, and relative novel-output rate, providing a method-level diagnostic instrument for evaluating thinking mode and epistemic bias during hazard analysis. A Novelty Test and a Residual Novelty Rate metric are proposed to assess the proportion of predicted situations that are both original and atypical, enabling quantitative comparison of analytical approaches.<br/><br/>AIC Systems Approach entails the following stages; Stage 1 converts a high-level operational brief into an exhaustively articulated complexity model through STCoT predictive queries, enabling systematic enumeration of both typical and atypical contributing situations. Stage 2 screens discovered scenarios to scope the design solution boundary. Stage 3 introduces the autonomous system into the problem complexity, models bidirectional AIC interactions, conducts HazTOPS (Hazards, Threats, Opportunities) analysis, and derives system, safety, and ML training requirements with full downward traceability. Stage 4 applies AIC perspective shifts, altering interaction type, direction, or effect polarity, to generate structurally justified Black Swan scenarios unreachable through conventional deviation analysis. Stage 5 converts ML safety requirements into CuneiForms: pictorial-type requirements specifying the Valuable Minimum Variety of visual configurations an ML model must encounter during training, closing the semantic gap between textual requirements and dataset content. Stage 6 implements a Black Swan-driven incremental ML development and testing pipeline in which dataset distributional novelty is validated through perceptual hash similarity testing and progressive Black Swan incorporation produces an auditable ALARP evidence trail.<br/><br/>The approach is evaluated through two independent case studies. In Case Study 1, applied to an unsafe train-tracks autonomous security drone problem, Stage 1 identified 133 contributing situations compared to 3 produced by informal pre-analysis. The Novelty Test classified 54 of these as novel and atypical Black Swan candidates, yielding a Novelty Rate of Ṅ= 0.41 (95% Wilson CI: [0.32, 0.49]) and a Residual Novelty Rate of 0.59, compared to an estimated baseline residual rate of 1.0 for HAZOP, STAMP, and FRAM applied to the same problem. In Case Study 2, applied to the AVOIDDS airborne object detection dataset, Stage 1 yielded Ṅ= 0.33 (95% Wilson CI: [0.24, 0.43]) and a Residual Novelty Rate of 0.67. Comparative ATD analysis suggests that the AIC approach produces a more balanced, long-tailed attention distribution with lower coverage blindness than HAZOP, STAMP, and FRAM applied to the same scenario.<br/><br/>The Stage 6 ML experiment demonstrates that a model trained without Black Swan data achieves 13% mAP@50 on Black Swan test sets despite 99% performance on typical operational data, providing direct empirical evidence that typical-operations validation alone produces a false sense of assurance. Progressive Black Swan incorporation recovers performance to 99% across all scenario types. Case Study 2 further demonstrates that retrospective CuneiForm analysis of an existing published dataset identifies coverage gaps not visible from the dataset's documentation alone, confirming the approach's applicability to dataset audit as well as dataset design.

Degree

thesis:*
Name dc:type.qualificationname
Ph.D.
Level dc:type.qualificationlevel
doctoral
Grantor dc:publisher.institution
University of Southampton
Year dc:date.issued
2026

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Al-Shareefy, Haider Mahmood
Advisors dc:contributor.advisor
  • Butler, Michael
  • Hoang, Son

Rights

Language dc:language
English, eng

Identifiers

dc:identifier.*

Chain of custody

source
Harvested from
University of Southampton
Base URL
eprints.soton.ac.uk/cgi/oai2
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
related terms
citation

Al-Shareefy, Haider Mahmood. Advances in hazard management for machine learning based safety-critical systems operating in unpredictable environments. doctoral thesis, University of Southampton, 2026. https://doi.org/10.5258/SOTON/PG/T303