Back to search

Boston University

Quantification of improvement in risk prediction models

Abstract

dc:description.abstract

The identification of new factors in modeling the probability of a binary outcome is the main challenge for statisticians and clinicians who want to improve risk prediction. This motivates researchers to search for measures that quantify the performance of new markers. There are many commonly used measures that assess the performance of the binary outcome model: logistic R-squares, discrimination slope. area under the ROC (receiver operating characteristic) curve (AUC) and Hosmer-Lemeshow goodness of fit chi-square. However, metrics that work well for model assessment may not be as good for quantifying the usefulness of new risk factors, especially when we add a new predictor to a well performing baseline model. The recently proposed new measures of improvement, the Integrated Discrimination Improvement (IDI) - difference between discrimination slopes - and the Net Reclassification Improvement (NRI), directly address the question of model performance and take it beyond the simple statistical significance of a new risk factor. Since these two measures are new and have not been studied as extensively as the older ones, a question of their interpretation naturally arises. In our research we propose meaningful interpretations to the new measures as well as extensions of these measures that address some of their potential shortcomings. Following the derivation of the maximum R-squared by Nagelkerke, we show how the IDI, which depends on the event rate, could be rescaled by its hypothetical maximum value to reduce this dependence. Furthermore, the IDI metric assumes a uniform distribution for all risk cut-offs. Application of clinically important thresholds prompted us to derive a metric that includes a prior distribution for the cut-off points and assigns different weights to sensitivity and specificity. Similarly, we propose the maximum and rescaled NRI. The latter is based on counting the number of categories by which the risk of a given person moved and guarantees that reclassification tables with equal marginal probabilities will lead to a zero NRI. All developments are investigated employing numerical simulations under the assumption of normality and varying effect sizes of the associations. We also illustrate the proposed concepts using examples from the Framingham Heart Study.

Degree

thesis:*
Name thesis:degree_name
Doctor of Philosophy
Level thesis:degree_level
doctoral
Discipline thesis:degree_discipline
Mathematics
Grantor dc:publisher
Boston University
Year dc:date.issued
2012

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Pencina, Karol M.

Rights

Language dc:language.iso
en_US

Identifiers

dc:identifier.*
Dc Identifier Barcode
11719032089312
Dc Identifier Bibid
99199980250001161

Chain of custody

source
Harvested from
Boston University
Base URL
open.bu.edu/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
related terms
citation

Pencina, Karol M.. Quantification of improvement in risk prediction models. doctoral thesis, Boston University, 2012. https://hdl.handle.net/2144/32045