Back to search

Michigan State University

Assessing the impact of missing data on hospital performance profiling

Abstract

dc:description

Ischemic stroke is a leading cause of mortality, long-term disability, and high healthcare costs in the US. In light of this clinical and financial burden, the Centers for Medicare & Medicaid Services (CMS) has decided to incorporate ischemic stroke measures of 30-day mortality and hospital readmission into its current pay-for-performance program. This decision has come under intense scrutiny, as many clinicians and researchers believe that the current risk adjustment model is inadequate because it does not include a measure of stroke severity. Due to its well-documented importance in individual-level prediction, there is concern that excluding a measure of stroke severity from risk adjustment will lead to incorrect rankings of hospital performance, i.e. hospital profiling. However, administrative datasets used in CMS currently do not capture a measure of stroke severity, such as the National Institutes of Health Stroke Scale (NIHSS), and in clinical databases which capture NIHSS, it is frequently missing. Little work has been done to asses if the documentation of NIHSS is biased, and if so, what impact bias would have on hospital-level estimates of mortality. In this study, we analyzed data from ischemic stroke patients from an existing stroke registry to identify patterns and characteristics that predict NIHSS documentation at the patient- and hospital-level. Next, we tested for the presence of selection bias in patients with documented NIHSS using the Heckman Selection Model. Finally, using computer simulations, we estimated the impact of missing NIHSS data on hospital profiling of 30-day mortality, under different assumptions about the prevalence and mechanism of missing NIHSS data. We found that patients with documented NIHSS were, in fact, a biased subsample of all ischemic stroke patients. Documentation of NIHSS was driven by a combination of patient-level and hospital-level factors. At the patient- and hospital-level, analyses suggested that patients with more severe strokes (i.e. increased NIHSS score) were better documented than patients with less severe strokes. These findings were confirmed using the Heckman Selection Model. However, in both analyses, we found that the amount of bias was modest. In computer simulations, we quantified the impact that missing data would have on the accuracy of hospital ischemic stroke profiling, under different assumptions about how NIHSS data was missing. Any effect of missing NIHSS mechanism was trumped by the impact of missingness on sample size. Because patients with missing NIHSS data were dropped from risk-adjustment models as documentation of NIHSS decreased, the accuracy of hospital risk-standardized mortality rates (RSMRs) estimated by the hierarchical logistic model deteriorated. All of our findings were substantially modified by the hospital ischemic stroke volume, with low volume hospital suffering the worst accuracy. These results are a reflection of the fact that the loss of sample size (either through the documentation rate or hospital volume), increases the amount of shrinkage in RSMR estimates, which makes any random noise more impactful on changes in RSMR. Overall, our findings raise concerns about the addition of NIHSS data into risk adjustment models for hospital-level ischemic stroke outcomes, and illustrate shortcomings in current methodologies used to profile hospitals. It is crucial that data used in risk adjustment for hospital profiling be documented with very high levels of completeness.

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Thompson, Michael P., Ph. D.
Contributors dc:contributor
  • Reeves, Mathew J.
  • Luo, Zhehui
  • Gardiner, Joseph
  • Burke, James F.

Subjects

dc:subject × 4

Rights

dc:rights
Statement dc:rights
  • In Copyright
Language dc:language
English

Identifiers

dc:identifier.*
Identifier
etd:2628
isbn:9781321717167
isbn:1321717164
oclc:933586556
umi:3701213
local:Thompson_grad.msu_0128D_13621

Chain of custody

source
Harvested from
Michigan State University
Base URL
d.lib.msu.edu/oai
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Thompson, Michael P., Ph. D.. Assessing the impact of missing data on hospital performance profiling. 2015. https://doi.org/doi:10.25335/3v9d-yq08