{"id":{"repo_id":"calpoly","oai_identifier":"oai:digitalcommons.calpoly.edu:theses-4890"},"canonical_url":"https://search.ndltd.org/etd/calpoly/oai:digitalcommons.calpoly.edu:theses-4890","repository":{"repo_id":"calpoly","name":"Cal Poly","base_url":"https://digitalcommons.calpoly.edu/do/oai/"},"display":{"title":"Using Ensemble Disagreement to Stabilize Conformal Prediction Under Distribution Shift","abstract":"Semantic segmentation of eelgrass from drone imagery is crucial for coastal habitat monitoring, restoration, and management, as these habitats continue to see rapid changes due to climate change and human influence. However, the reliability of generalizing a deployed classification model relies on both high-accuracy segmentation as well as robust uncertainty quantification that holds up when conditions change over years or locations. Conformal prediction (CP) is a method that converts a classifier's output into prediction sets with a guaranteed average coverage level for in-distribution data. However, the “vanilla” conformal score can often under-cover in hard or out-of-distribution (OOD) regions under drift. Motivated by difficulty-adaptive conformal approaches, this study investigates normalizing conformal scores by a proxy for epistemic uncertainty, ensemble disagreement. Nonconformity scores are rescaled by an estimated metric of local difficulty learned on the calibration set, so that the global quantile is conservative where OOD risk is high. This study evaluates two normalization methods: a parametric linear normalization and a nonparametric normalization. This creates a locally adaptive score. On drone imagery of the Morro Bay estuary, California (2018-2022), the models were trained on 2018-2021, calibrated on 2021, and evaluated on 2022 on OOD points. Normalized scores recover all or almost all of the lost coverage seen by vanilla split conformal prediction, increase the percentage of singletons, and reduce the variability in spatial coverage. In this work, we provide an easy-to-implement drop-in framework for regaining coverage in challenging regions and out-of-distribution data, eliminating the need for retraining or labels at test time.","abstract_html":"Semantic segmentation of eelgrass from drone imagery is crucial for coastal habitat monitoring, restoration, and management, as these habitats continue to see rapid changes due to climate change and human influence. However, the reliability of generalizing a deployed classification model relies on both high-accuracy segmentation as well as robust uncertainty quantification that holds up when conditions change over years or locations. Conformal prediction (CP) is a method that converts a classifier&#x27;s output into prediction sets with a guaranteed average coverage level for in-distribution data. However, the “vanilla” conformal score can often under-cover in hard or out-of-distribution (OOD) regions under drift. Motivated by difficulty-adaptive conformal approaches, this study investigates normalizing conformal scores by a proxy for epistemic uncertainty, ensemble disagreement. Nonconformity scores are rescaled by an estimated metric of local difficulty learned on the calibration set, so that the global quantile is conservative where OOD risk is high. This study evaluates two normalization methods: a parametric linear normalization and a nonparametric normalization. This creates a locally adaptive score. On drone imagery of the Morro Bay estuary, California (2018-2022), the models were trained on 2018-2021, calibrated on 2021, and evaluated on 2022 on OOD points. Normalized scores recover all or almost all of the lost coverage seen by vanilla split conformal prediction, increase the percentage of singletons, and reduce the variability in spatial coverage. In this work, we provide an easy-to-implement drop-in framework for regaining coverage in challenging regions and out-of-distribution data, eliminating the need for retraining or labels at test time.","abstract_has_math":false,"creators":["Murphy, Patrick D"],"institution":null,"degree_name":"MS in Statistics","degree_level":null,"degree_discipline":"Statistics","degree_department":null,"school":null,"contributors":["Kelly Bodwin","Statistics","College of Science and Mathematics"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":null,"date_issued":"","date_published":null,"updated_at":"2026-11-19T07:12:51Z","subjects":["Conformal prediction","adaptive sets","uncertainty","eelgrass","remote sensing","Applied Statistics","Artificial Intelligence and Robotics","Data Science","Environmental Monitoring"],"languages":[],"rights":[],"rights_urls":[],"identifier_entries":[],"person_identifier_entries":[]},"links":{"outbound_url":"https://digitalcommons.calpoly.edu/theses/3231","outbound_label":"Repository record","outbound_source":"dc:identifier","record_url":"https://digitalcommons.calpoly.edu/theses/3231","record_label":"Repository record","record_kind":"record","record_source":"dc:identifier","download_url":null,"download_label":null,"download_source":null,"access_status":null,"embargo_until":null,"access_note":null},"source_record":{"url":"https://digitalcommons.calpoly.edu/do/oai/?verb=GetRecord&metadataPrefix=oai_etdms&identifier=oai%3Adigitalcommons.calpoly.edu%3Atheses-4890","prefix":"oai_etdms"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Kelly Bodwin","Statistics","College of Science and Mathematics"]},{"key":"dc:creator","label":"Author","values":["Murphy, Patrick D"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date.available","label":"Dc Date Available","values":["2026-03-13T07:00:00Z"]},{"key":"dc:date.created","label":"Dc Date Created","values":["2026-03-01T08:00:00Z"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Statistics"]},{"key":"thesis:degree_name","label":"Degree Name","values":["MS in Statistics"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Conformal prediction","adaptive sets","uncertainty","eelgrass","remote sensing","Applied Statistics","Artificial Intelligence and Robotics","Data Science","Environmental Monitoring"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["https://digitalcommons.calpoly.edu/theses/3231"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description.abstract","label":"Abstract","values":["Semantic segmentation of eelgrass from drone imagery is crucial for coastal habitat monitoring, restoration, and management, as these habitats continue to see rapid changes due to climate change and human influence. However, the reliability of generalizing a deployed classification model relies on both high-accuracy segmentation as well as robust uncertainty quantification that holds up when conditions change over years or locations. Conformal prediction (CP) is a method that converts a classifier's output into prediction sets with a guaranteed average coverage level for in-distribution data. However, the “vanilla” conformal score can often under-cover in hard or out-of-distribution (OOD) regions under drift. Motivated by difficulty-adaptive conformal approaches, this study investigates normalizing conformal scores by a proxy for epistemic uncertainty, ensemble disagreement. Nonconformity scores are rescaled by an estimated metric of local difficulty learned on the calibration set, so that the global quantile is conservative where OOD risk is high. This study evaluates two normalization methods: a parametric linear normalization and a nonparametric normalization. This creates a locally adaptive score. On drone imagery of the Morro Bay estuary, California (2018-2022), the models were trained on 2018-2021, calibrated on 2021, and evaluated on 2022 on OOD points. Normalized scores recover all or almost all of the lost coverage seen by vanilla split conformal prediction, increase the percentage of singletons, and reduce the variability in spatial coverage. In this work, we provide an easy-to-implement drop-in framework for regaining coverage in challenging regions and out-of-distribution data, eliminating the need for retraining or labels at test time."]},{"key":"dc:title","label":"Title","values":["Using Ensemble Disagreement to Stabilize Conformal Prediction Under Distribution Shift"]}]}],"canonical_facts":{"dc:contributor":["Kelly Bodwin","Statistics","College of Science and Mathematics"],"dc:creator":["Murphy, Patrick D"],"dc:date.available":["2026-03-13T07:00:00Z"],"dc:date.created":["2026-03-01T08:00:00Z"],"dc:description.abstract":["Semantic segmentation of eelgrass from drone imagery is crucial for coastal habitat monitoring, restoration, and management, as these habitats continue to see rapid changes due to climate change and human influence. However, the reliability of generalizing a deployed classification model relies on both high-accuracy segmentation as well as robust uncertainty quantification that holds up when conditions change over years or locations. Conformal prediction (CP) is a method that converts a classifier's output into prediction sets with a guaranteed average coverage level for in-distribution data. However, the “vanilla” conformal score can often under-cover in hard or out-of-distribution (OOD) regions under drift. Motivated by difficulty-adaptive conformal approaches, this study investigates normalizing conformal scores by a proxy for epistemic uncertainty, ensemble disagreement. Nonconformity scores are rescaled by an estimated metric of local difficulty learned on the calibration set, so that the global quantile is conservative where OOD risk is high. This study evaluates two normalization methods: a parametric linear normalization and a nonparametric normalization. This creates a locally adaptive score. On drone imagery of the Morro Bay estuary, California (2018-2022), the models were trained on 2018-2021, calibrated on 2021, and evaluated on 2022 on OOD points. Normalized scores recover all or almost all of the lost coverage seen by vanilla split conformal prediction, increase the percentage of singletons, and reduce the variability in spatial coverage. In this work, we provide an easy-to-implement drop-in framework for regaining coverage in challenging regions and out-of-distribution data, eliminating the need for retraining or labels at test time."],"dc:identifier":["https://digitalcommons.calpoly.edu/theses/3231"],"dc:subject":["Conformal prediction","adaptive sets","uncertainty","eelgrass","remote sensing","Applied Statistics","Artificial Intelligence and Robotics","Data Science","Environmental Monitoring"],"dc:title":["Using Ensemble Disagreement to Stabilize Conformal Prediction Under Distribution Shift"],"thesis:degree_discipline":["Statistics"],"thesis:degree_name":["MS in Statistics"]},"updated_at":"2026-11-19T07:12:51Z"}