My objective was to predict outcomes following police interactions with PMIs, and compare the predictive accuracy of logistic regression models and Random Forests learning algorithms. Additionally I evaluated if predictive accuracy of Random Forests changed when applied to merged versus region-specific data. I conducted a retrospective cohort study of reports completed by police in 13 communities between 2015 and 2018. 13,058 reports were analyzed. Random Forests learning algorithms were compared against logistic regression models for predictive accuracy in a merged dataset (13 communities) and 3 regional datasets. Outcomes for prediction were high risk of harm to self, risk of harm to others, and risk of failure to care for self within 24 and 72 hours following police contact. Random Forests learning algorithms were trained on merged and regional datasets, and compared against merged and regional holdout datasets. Performance was compared by area under the curve. For Random Forests learning algorithms, confusion matrix statistics were calculated for each outcome and predictive utility was examined by calculating conditional probabilities.
Prediction accuracy was modest across all methods. Random Forests achieved better predictive accuracy than logistic regression. Random Forests accuracy varied between merged and regional holdout data. Sensitivity of Random Forests learning algorithms were moderate (74% average, 6 outcomes, merged holdout set). Specificity was low (53% average, 6 outcomes, merged holdout set). Conditional probabilities were modestly improved by the use of the Random Forests learning algorithm. The rareness of the target outcomes created a situation where even predictions with moderate likelihood ratios had only modest predictive value. Though the Random Forests learning algorithms did outperform the logistic regression learning algorithms, the clinical significance of those benefits were limited when conditional probabilities were calculated. These findings are limited to the outcomes considered, and may not apply to more common outcomes. / Thesis / Master of Health Sciences (MSc) / The study goal was to predict outcomes following police interactions with persons with mental illness (PMIs). Additionally we compare the predictive validity of logistic regression and Random Forests learning algorithms. Classification approaches were applied to outcomes following police interactions with PMIs, including: high risk of harm to self, high risk of harm to others, and high risk of failure to care for self within 24 hours and 72 hours of initial police contact. The study also sought to determine if the predictive accuracy of Random Forests was sensitive to the police service community. Variation in predictive accuracy was assessed between a merged data set (13 communities) and 3 community-specific data. The study found that the predictive accuracy of the classification approaches on outcomes was modest. Random Forests exhibited greater predictive validity than logistic regression. The performance of the Random Forests suggested that performance was not sensitive to police service context.
| Identifer | oai:union.ndltd.org:mcmaster.ca/oai:macsphere.mcmaster.ca:11375/24476 |
| Date | January 2019 |
| Creators | Pearson Hirdes, Daniel |
| Contributors | Costa, Andrew, eHealth |
| Source Sets | McMaster University |
| Language | English |
| Detected Language | English |
| Type | Thesis |
Page generated in 0.0019 seconds