Back to search

University of British Columbia

Efficient online uncertainty management : Conformal-in-the-Loop for measuring and actioning uncertainty at training time

Abstract

dc:description

Focus on Neural Network research has largely been on statistical performance, as Deep Learning is an optimization problem. This often overshadows properties needed for adoption, such as robustness and fairness. Early adversarial attacks showed minor input changes could cause incorrect predictions. Fairness is obscured by the black-box nature of neural networks, which are typically evaluated by a single performance statistic. This thesis explores uncertainty quantification motivated by improving fairness and robustness. A use case on fair artificial intelligence (AI) explores the Breast Screening Program at the BC Cancer Agency with the evaluation of a commercial AI. Model performance was evaluated using the area under the curve of receiver-operating characteristic. Compared to radiologists, it underperforms on follow-up periods up to two years but overperforms for longer periods. It performs well overall on a Canadian population, but performs poorly in certain subgroups including BI-RADS densities, patients with a self-reported family history of cancer, and mammograms showing calcifications. A second use case evaluates a car detection algorithm using the KITTI dataset, where the model shows a lack of robustness for small perturbations within sensor error. These use cases motivate the thesis. Uncertainty is evaluated by comparing the performance and diversity of 144 models from a grid search of network parameters for semantic segmentation of the CityScapes dataset. Many models show no significant performance difference, and performance and diversity across models can be predicted. This diversity can be used a measure of uncertainty in model ensembles. Uncertainty quantification during training is often used for online labeling and pruning mislabeled examples. However, these methods are usually computationally expensive and require multiple stages or training runs. The final chapter applies conformal prediction for efficient online data curation in a single training run. It proposes a data-centric method for dynamically weighting examples using prediction set size. Evaluated on CIFAR-10 for classification and CityScapes for segmentation, results show improved accuracy and improved Jaccard Index with small computational overhead.

Degree

thesis:*
Name thesis:degree_name
Doctor of Philosophy - PhD
Level thesis:degree_level
doctoral
Discipline thesis:degree_discipline
Computer Science
Grantor dc:publisher
University of British Columbia
Year dc:date
2025

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Graham-Knight, John Brandon
Contributors dc:contributor
  • Lasserre, P.
  • Najjaran, Homayoun

Rights

dc:rights
Statement dc:rights
  • Attribution-NonCommercial-ShareAlike 4.0 International
Language dc:language
eng

Identifiers

dc:identifier.*
Handle dc:identifier
http://hdl.handle.net/2429/91243

Chain of custody

source
Harvested from
University of British Columbia
Base URL
circle.library.ubc.ca/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
related terms
citation

Graham-Knight, John Brandon. Efficient online uncertainty management : Conformal-in-the-Loop for measuring and actioning uncertainty at training time. doctoral thesis, University of British Columbia, 2025. http://hdl.handle.net/2429/91243