Integrating personalized risk predictions into clinical decision support requires well-calibrated models, yet model accuracy deteriorates as patient populations shift. Understanding the influence of modeling methods on performance drift is essential for designing updating protocols. Using national cohorts of Department of Veterans Affairs hospital admissions, we compared the temporal performance of seven regression and machine learning models for hospital-acquired acute kidney injury and 30-day mortality after admission. All modeling methods were robust in terms of discrimination and experienced deteriorating calibration. Random forest and neural network models experienced lower levels of calibration drift than regressions. The L-2 penalized logistic regression for mortality demonstrated drift similar to the random forest. Increasing overprediction by all models correlated with declining event rates. Diverging patterns of calibration drift among acute kidney injury models coincided with predictor-outcome association changes. The mortality models revealed reduced susceptibility of random forest, neural network, and L-2 penalized logistic regression models to case mix-driven calibration drift. These findings support the advancement of clinical predictive analytics and lay a foundation for systems to maintain model accuracy. As calibration drift impacted each method, all clinical prediction models should be routinely reassessed and updated as needed. Regression models have a greater need for frequent evaluation and updating than machine learning models, highlighting the importance of tailoring updating protocols to variations in the susceptibility of models to patient population shifts. While the suite of best practices remains to be developed, modeling methods will be an essential component in determining when and how models are updated.
| Identifer | oai:union.ndltd.org:VANDERBILT/oai:VANDERBILTETD:etd-03272017-091807 |
| Date | 05 April 2017 |
| Creators | Davis, Sharon Elizabeth |
| Contributors | Michael E Matheny, Thomas A Lasko, Guanhua Chen |
| Publisher | VANDERBILT |
| Source Sets | Vanderbilt University Theses |
| Language | English |
| Detected Language | English |
| Type | text |
| Format | application/pdf |
| Source | http://etd.library.vanderbilt.edu/available/etd-03272017-091807/ |
| Rights | restricted, I hereby certify that, if appropriate, I have obtained and attached hereto a written permission statement from the owner(s) of each third party copyrighted matter to be included in my thesis, dissertation, or project report, allowing distribution as specified below. I certify that the version I submitted is the same as that approved by my advisory committee. I hereby grant to Vanderbilt University or its agents the non-exclusive license to archive and make accessible, under the conditions specified below, my thesis, dissertation, or project report in whole or in part in all forms of media, now or hereafter known. I retain all other ownership rights to the copyright of the thesis, dissertation or project report. I also retain the right to use in future works (such as articles or books) all or part of this thesis, dissertation, or project report. |
Page generated in 0.0029 seconds