Introduction: A protocol for management of sedation and pain for mechanically ventilated patients at Flagstaff Medical Center (FMC) was implemented in August 2013. It was unknown whether the protocol is being adhered to or whether it has had an impact on patient outcomes. Objectives: To develop an audit and feedback mechanism to monitor adherence to sedation protocol at FMC and determine whether the protocol has impacted patient outcomes. Methods: A retrospective manual chart review was conducted including all mechanically ventilated adult patients for four, one-month periods: 1) pre-protocol; and 2) one month, 3) six months, and 4) 12 months post-protocol implementation. Results: 132 total patients were included (32 pre; 100 post-protocol). Mean weighted adherence score for post-protocol study groups were 5.0±0.6, 5.0±0.7, and 5.2±0.7 (p=0.926) out of ten. Time of mechanical ventilation (p=0.003) and hospital length of stay (LOS) (p=0.023) were reduced post (56±58h; 9.8±7.9days) vs. pre-protocol (90±67h; 13±7days). The adherence score was weakly correlated with hospital LOS but not time of mechanical ventilation. Conclusion: This project demonstrates improvements in patient outcomes from utilization of a sedation protocol. However, this project also highlights several challenges associated with the monitoring of protocol adherence. A lack of audit and feedback may be a factor in the observed unchanged adherence over time. Both research and monitoring activities are impaired by EHR systems that do not allow for the easy extraction of data. Ensuring that adequate audit and feedback strategies are designed and available prior to implementation of new protocols is an essential step in planning the implementation of a new protocol.

BACKGROUND: Audit and feedback interventions have the potential to improve clinical care. Electronically captured administrative and clinical data routinely collected in Canadian hospitals may be used to provide feedback to physicians in general medicine in-patient care. The computation of appropriate quality indicator requires patient care to be attributed to individual physician(s). The appropriate attribution method in contexts where multiple physicians are involved in the care with varying degree of responsibilities that change over time is not straight forward. There has so far been little guidance in the literature of how to best accomplish this. The objective of this study is to identify trade-offs of different physician attribution methods by applying them to the same large clinical dataset. METHODS: A retrospective cohort study was conducted using the GEMINI dataset consisting of administrative and clinical data of hospitalized patients discharged from General Medicine service between April 1, 2010 and October 31, 2017 extracted from electronic systems at 7 hospitals in the Greater Toronto Area. A set of four quality indicators (length of stay, 30-day re-admission, in-patient mortality, use of advanced imaging) used in an audit and feedback intervention was calculated for each physician using 5 different physician attribution methods: STRICT (only patients with the same admitting, discharging, and most responsible physician with length of stay less than 14 days were included to capture those patients whose care was provided by only 1 physician), ADMIT (attribute care to admitting physician), DISCHARGE (attribute care to discharging physician), MRP (attribute care to most responsible physician), and ANY (attribute care to admitting, discharging, and most responsible physicians). The comprehensiveness and comparability of each attribution method were calculated. The actual differences of the indicator value and physician ranking for each indicator was compared between each pair of attribution methods. RESULTS: 222,490 hospitalization cared for by 203 physicians were included. STRICT attribution method was least comprehensive, capturing only 40% of patients cared for by a physician), while ADMIT, DISCHARGE, and MRP captured 70% of patients. All attribution methods produced patient populations for individual physicians that were comparable to those seen at each hospital. STRICT attribution method resulted in length of stay values 4.7 to 6.8 days shorter than other attribution methods and had poor rank correlation of physicians when compared to other attribution methods (spearman rank correlation 0.27 to 0.52). Absolute differences for the other 3 indicators were small between all attribution methods, and relative ranking of physicians were reasonably preserved (strong or very strong rank correlation). INTERPRETATION: Different attribution methods have different comprehensiveness, but all produced mostly comparable patient populations for physicians. Certain attribution method can affect apparent physician performance for some quality indicators but not others. The impact of physician attribution methods deserve consideration during the design of audit and feedback interventions. / Graduate

Background: UK primary care accounts for 90% of patient contacts in the NHS, and over 300 million consultations every year. Consequently, when primary is suboptimal it has important impacts on population health. At the same time, virtually all general practices use electronic health records (EHR) to capture patient data. Clinical Decision Support (CDS) systems use it to highlight when individual patients do not receive care consistent with clinical guidelines, though ignore the wider population. Electronic Audit and Feedback (e-A&F) systems address the wider population, but their results are difficult to interpret. EHR data has the richness to suggest ways in which care quality could be improved, though this is currently not exploited. The aim of this thesis was to make progress towards better use of primary care EHR data for the purposes of quality improvement (QI) by focusing on e-A&F as a vehicle. Research Objectives were: 1) Develop a model and recommendations to guide EHR data analysis and its communication to health professionals; 2) Use these models and recommendations to develop a system for UK primary care; 3) Implement and evaluate the system to test the models and recommendations, and derive generalisable knowledge. Methods: The overall approach of this thesis was informed by guidance from the Medical Research Council on the development of complex interventions, and Boyrcki et al.’s evidence-based framework for the development of health information technologies (Chapter 2). Theory was first identified through a critical examination of the empirical and theoretical literature regarding CDS and e-A&F systems (Chapter 3), then built upon in a systematic literature search and metasynthesis of qualitative studies of A&F (and e-A&F) interventions (Chapter 4). This resulted in the development a new theory of A&F (Clinical Performance Feedback Intervention Theory; CP-FIT), which was used to inform the development of an e-A&F system for UK primary care – the Performance Improvement plaN GeneratoR (PINGR; version 1). PINGR was then iteratively optimised through a series of three empirical studies. First, its usability was evaluated by software experts using Heuristic Evaluation and Cognitive Walkthrough methodologies (Chapter 5). GPs then performed structured tasks using the system in a laboratory whilst their on-screen interactions and eye movements were recorded (Chapter 6). Finally, PINGR was implemented in 15 GP practices, and CP-FIT used to guide the mixed methods evaluation including examinations of usage records, and interviews with 38 health professionals. Results: There are both empirical and theoretical arguments for combining features from CDS and e-A&F systems to increase their effectiveness; a key recommendation is that e-A&F systems should suggest clinical actions to health professionals (Chapter 3). This is supported by CP-FIT, which has three core propositions: 1) A&F interventions exert their effects through health professionals taking action; 2) Health care organisations have limited capacity to engage with A&F; and 3) Health care professionals and organisations have a strong set of beliefs and behaviours regarding how they provide patient care (Chapter 4). Based on these findings, the unique feature of PINGR is that it suggests improvement actions to users based on EHR data analysis (‘decision-supported feedback’). Key findings from PINGR’s usability evaluation with software experts translated into a set of design guidelines for e-A&F interfaces regarding: summarising clinical performance, patient lists, patient-level information, and suggested actions (Chapter 5). When tested with GPs, these guidelines were found to impact: user engagement; actionability; and information prioritisation (Chapter 6). Following its implementation in practice, PINGR was used on 227 occasions to facilitate the care of 725 patients. These patients were 1.8 (95% CI 1.6-1.9) times more likely to receive improved care according to at least one clinical guideline. Barriers and facilitators to its success included: the resources available to use it; its perceived relative advantages; how compatible it was with pre-existing beliefs and ways of working; the credibility of its data; the complexity of the clinical problems it highlighted; and the ability to act on its recommendations (Chapter 7). Conclusion: It is both feasible and acceptable to health professionals to make better use of EHR data for QI by enabling e-A&F systems to suggest actions for them to take. When designing e-A&F interfaces, attention should be paid to how they summarise clinical performance, and present patient lists and detailed patient-level information. Implementation of e-A&F interventions is influenced by availability of resources, compatibility with existing workflows, and ability to take action based on their feedback results. Unresolved tensions exist regarding how they may deal with patient complexity. Policymakers should consider the relevance of these findings for National Clinical Audits and pay-for-performance initiatives.

Yes / Background: Interpreting evaluations of complex interventions can be difficult without sufficient description of key intervention content. We aimed to develop an implementation package for primary care which could be delivered using typically available resources and could be adapted to target determinants of behaviour for each of four quality indicators: diabetes control, blood pressure control, anticoagulation for atrial fibrillation and risky prescribing. We describe the development and prospective verification of behaviour change techniques (BCTs) embedded within the adaptable implementation packages. Methods: We used an over-lapping multi-staged process. We identified evidence-based, candidate delivery mechanisms—mainly audit and feedback, educational outreach and computerised prompts and reminders. We drew upon interviews with primary care professionals using the Theoretical Domains Framework to explore likely determinants of adherence to quality indicators. We linked determinants to candidate BCTs. With input from stakeholder panels, we prioritised likely determinants and intervention content prior to piloting the implementation packages. Our content analysis assessed the extent to which embedded BCTs could be identified within the packages and compared them across the delivery mechanisms and four quality indicators. Results: Each implementation package included at least 27 out of 30 potentially applicable BCTs representing 15 of 16 BCT categories. Whilst 23 BCTs were shared across all four implementation packages (e.g. BCTs relating to feedback and comparing behaviour), some BCTs were unique to certain delivery mechanisms (e.g. ‘graded tasks’ and ‘problem solving’ for educational outreach). BCTs addressing the determinants ‘environmental context’ and ‘social and professional roles’ (e.g. ‘restructuring the social and ‘physical environment’ and ‘adding objects to the environment’) were indicator specific. We found it challenging to operationalise BCTs targeting ‘environmental context’, ‘social influences’ and ‘social and professional roles’ within our chosen delivery mechanisms. Conclusion: We have demonstrated a transparent process for selecting, operationalising and verifying the BCT content in implementation packages adapted to target four quality indicators in primary care. There was considerable overlap in BCTs identified across the four indicators suggesting core BCTs can be embedded and verified within delivery mechanisms commonly available to primary care. Whilst feedback reports can include a wide range of BCTs, computerised prompts can deliver BCTs at the time of decision making, and educational outreach can allow for flexibility and individual tailoring in delivery / UK National Institute for Health Research (NIHR) under its Programme Grants for Applied Research scheme (Grant Reference Number RP-PG-1209-10040).

A demanda por procedimentos médicos gerados pela transformação dos perfis etários e de morbi-mortalidade das populações tem forçado uma escalada contínua dos custos da assistência que atingem limites críticos em relação à disponibilidade de recursos. É neste contexto onde a priorização é a regra e o desperdício inaceitável, que métodos de otimização vem sendo desenvolvidos, dentre eles o “Audit and feedback”. De um total de 176 médicos de uma regional de cooperativa, 130 constituíram a amostra para estudo Quase-experimento com total de dois anos de seguimento (2004- 2005). As informações tratavam do “Índice de Solicitações de Serviços Auxiliares de Diagnose e Terapia” (Is SADT) e eram fornecidas de modo que cada profissional sabia qual era a sua posição dentro do grupo da mesma especialidade, mas não tinha condições de identificar os demais. Apesar do número absoluto de SADT realizados ter sido no mínimo o quádruplo do esperado para a população de beneficiários, o retorno de informações visando à conscientização pelo método “Audit and Feedback” não foi capaz de provocar diferenças significas nas solicitações de procedimentos nos diferentes grupos de médicos, conforme constatado após análise estatística. Não se evidenciou nenhuma relação significativa das variáveis independentes deste estudo (sexo, tempo de formado e tempo de cooperado), com os Índices de Solicitações de SADT dos médicos que estavam acima ou abaixo da mediana nos diferentes estratos. Persiste como desafio, a comprovação do fator ou fatores responsáveis por esta resposta. Propõe-se um conjunto de ações a serem associadas, uma vez que o processo de conscientização isoladamente foi inócuo. / Changing age, morbidity and mortality profiles of populations have caused an increase in the demand for medical procedures and a continuous increase in the cost of health services that is reaching limits in terms of available resources. It is in the context of prioritization as the rule and waste as unacceptable that optimization methods have been developed, among them “audit and feedback”. Of a total of 176 physicians of a regional cooperative, 130 were included in the study sample. The procedure ordering behaviour of each physician in the sample was monitored during all of 2004 and 2005. Audit: each month the an “index of solicitations for auxiliary diagnostic and treatment services”, or SADT, basically the average number of orders per consultation for the month, was calculated for each physician. Feedback: SADT for all participating physicians, organized by speciality and rank, were sent to each participating physician. The results were presented in a way that each physician could locate themselves in the ranking without being able to identify where any other physicians placed in the ranking. Despite absolute number of SADT on average fourfold that expected for the demographic profile of the population being served, statistical analysis of the results showed that the “Audit and Feedback” optimization method did not have a significant effect on the ordering behaviour of physicians or in the sample, un-grouped or grouped by speciality, gender, time in practice or time in the cooperative. Nor did grouping according the whether and individual physician was initially above or below the median SADT reveal any significant effect of the method on ordering behaviour. The results do show that factors other than how a physician's individual SADT compares to that of his or her peers, determines their procedure ordering behaviour. Identifying these factors remains an important challenge in the context of health services optimization.

A demanda por procedimentos médicos gerados pela transformação dos perfis etários e de morbi-mortalidade das populações tem forçado uma escalada contínua dos custos da assistência que atingem limites críticos em relação à disponibilidade de recursos. É neste contexto onde a priorização é a regra e o desperdício inaceitável, que métodos de otimização vem sendo desenvolvidos, dentre eles o “Audit and feedback”. De um total de 176 médicos de uma regional de cooperativa, 130 constituíram a amostra para estudo Quase-experimento com total de dois anos de seguimento (2004- 2005). As informações tratavam do “Índice de Solicitações de Serviços Auxiliares de Diagnose e Terapia” (Is SADT) e eram fornecidas de modo que cada profissional sabia qual era a sua posição dentro do grupo da mesma especialidade, mas não tinha condições de identificar os demais. Apesar do número absoluto de SADT realizados ter sido no mínimo o quádruplo do esperado para a população de beneficiários, o retorno de informações visando à conscientização pelo método “Audit and Feedback” não foi capaz de provocar diferenças significas nas solicitações de procedimentos nos diferentes grupos de médicos, conforme constatado após análise estatística. Não se evidenciou nenhuma relação significativa das variáveis independentes deste estudo (sexo, tempo de formado e tempo de cooperado), com os Índices de Solicitações de SADT dos médicos que estavam acima ou abaixo da mediana nos diferentes estratos. Persiste como desafio, a comprovação do fator ou fatores responsáveis por esta resposta. Propõe-se um conjunto de ações a serem associadas, uma vez que o processo de conscientização isoladamente foi inócuo. / Changing age, morbidity and mortality profiles of populations have caused an increase in the demand for medical procedures and a continuous increase in the cost of health services that is reaching limits in terms of available resources. It is in the context of prioritization as the rule and waste as unacceptable that optimization methods have been developed, among them “audit and feedback”. Of a total of 176 physicians of a regional cooperative, 130 were included in the study sample. The procedure ordering behaviour of each physician in the sample was monitored during all of 2004 and 2005. Audit: each month the an “index of solicitations for auxiliary diagnostic and treatment services”, or SADT, basically the average number of orders per consultation for the month, was calculated for each physician. Feedback: SADT for all participating physicians, organized by speciality and rank, were sent to each participating physician. The results were presented in a way that each physician could locate themselves in the ranking without being able to identify where any other physicians placed in the ranking. Despite absolute number of SADT on average fourfold that expected for the demographic profile of the population being served, statistical analysis of the results showed that the “Audit and Feedback” optimization method did not have a significant effect on the ordering behaviour of physicians or in the sample, un-grouped or grouped by speciality, gender, time in practice or time in the cooperative. Nor did grouping according the whether and individual physician was initially above or below the median SADT reveal any significant effect of the method on ordering behaviour. The results do show that factors other than how a physician's individual SADT compares to that of his or her peers, determines their procedure ordering behaviour. Identifying these factors remains an important challenge in the context of health services optimization.

A demanda por procedimentos médicos gerados pela transformação dos perfis etários e de morbi-mortalidade das populações tem forçado uma escalada contínua dos custos da assistência que atingem limites críticos em relação à disponibilidade de recursos. É neste contexto onde a priorização é a regra e o desperdício inaceitável, que métodos de otimização vem sendo desenvolvidos, dentre eles o “Audit and feedback”. De um total de 176 médicos de uma regional de cooperativa, 130 constituíram a amostra para estudo Quase-experimento com total de dois anos de seguimento (2004- 2005). As informações tratavam do “Índice de Solicitações de Serviços Auxiliares de Diagnose e Terapia” (Is SADT) e eram fornecidas de modo que cada profissional sabia qual era a sua posição dentro do grupo da mesma especialidade, mas não tinha condições de identificar os demais. Apesar do número absoluto de SADT realizados ter sido no mínimo o quádruplo do esperado para a população de beneficiários, o retorno de informações visando à conscientização pelo método “Audit and Feedback” não foi capaz de provocar diferenças significas nas solicitações de procedimentos nos diferentes grupos de médicos, conforme constatado após análise estatística. Não se evidenciou nenhuma relação significativa das variáveis independentes deste estudo (sexo, tempo de formado e tempo de cooperado), com os Índices de Solicitações de SADT dos médicos que estavam acima ou abaixo da mediana nos diferentes estratos. Persiste como desafio, a comprovação do fator ou fatores responsáveis por esta resposta. Propõe-se um conjunto de ações a serem associadas, uma vez que o processo de conscientização isoladamente foi inócuo. / Changing age, morbidity and mortality profiles of populations have caused an increase in the demand for medical procedures and a continuous increase in the cost of health services that is reaching limits in terms of available resources. It is in the context of prioritization as the rule and waste as unacceptable that optimization methods have been developed, among them “audit and feedback”. Of a total of 176 physicians of a regional cooperative, 130 were included in the study sample. The procedure ordering behaviour of each physician in the sample was monitored during all of 2004 and 2005. Audit: each month the an “index of solicitations for auxiliary diagnostic and treatment services”, or SADT, basically the average number of orders per consultation for the month, was calculated for each physician. Feedback: SADT for all participating physicians, organized by speciality and rank, were sent to each participating physician. The results were presented in a way that each physician could locate themselves in the ranking without being able to identify where any other physicians placed in the ranking. Despite absolute number of SADT on average fourfold that expected for the demographic profile of the population being served, statistical analysis of the results showed that the “Audit and Feedback” optimization method did not have a significant effect on the ordering behaviour of physicians or in the sample, un-grouped or grouped by speciality, gender, time in practice or time in the cooperative. Nor did grouping according the whether and individual physician was initially above or below the median SADT reveal any significant effect of the method on ordering behaviour. The results do show that factors other than how a physician's individual SADT compares to that of his or her peers, determines their procedure ordering behaviour. Identifying these factors remains an important challenge in the context of health services optimization.

Cette thèse, rédigée par articles, vise à examiner d’un point de vue pratique et théorique les enjeux sous-jacents à la capacité des infirmières d’agir sur leur pratique et son contexte par l’entremise d’interventions d’audit-feedback. L’audit-feedback est couramment utilisé dans le domaine de la santé en soutien aux changements et à l’amélioration des pratiques professionnelles. Bien que les infirmières soient régulièrement impliquées dans ces interventions, celles-ci sont étudiées surtout auprès des médecins, tant d’un point de vue empirique que théorique. Le contexte de pratique des infirmières comporte des caractéristiques peu facilitatrices à l’efficacité de l’audit-feedback. La méthodologie de développement théorique d’Alvesson et Kärreman (2007, 2011) a été utilisée afin de comprendre comment ces interventions pouvaient permettre aux infirmières d’agir sur leur pratique et son contexte. Le premier volet de la thèse implique une analyse de deux sources de données empiriques. D’abord, une revue systématique mixte a été effectuée pour comprendre les effets mesurés et perçus de l’audit-feedback auprès des infirmières. Les données des 13 études quantitatives suggèrent une efficacité très variable de l'audit-feedback. Les caractéristiques de la plupart des interventions étaient peu conformes aux recommandations actuelles dans le domaine. Globalement, les données des 18 études qualitatives suggèrent que les infirmières perçoivent plusieurs aspects négatifs à l’audit-feedback, tout en reconnaissant la pertinence d’une utilisation secondaire des données pour soutenir l'amélioration des soins. La seconde source de données provient d’une étude pilote sur le développement, la mise à l’essai et l’évaluation d’une intervention d’audit-feedback auprès d’une équipe de soins infirmiers en première ligne. Des indicateurs relatifs aux soins de plaies et mesurés à partir du système d’information I-CLSC ont été rapportés à l’équipe lors de deux séances de rétroaction. Des analyses de régression logistique ont été effectuées sur une période de 24 mois à partir de 1605 épisodes de soins de plaies afin d’évaluer l’évolution des indicateurs avant, pendant et après la période d’intervention. Un seul indicateur, celui de soutien à l’auto-gestion, a démontré une amélioration constante. L’analyse de ces données empiriques en fonction d’un cadre théorique spécifique à l’audit-feedback (Brown et al., 2019) a permis dans le second volet de la thèse de formuler trois hypothèses sur la réponse des infirmières à l’audit-feedback. Les hypothèses impliquent 1) une intégration de l’aspect relationnel de la pratique infirmière à la rétroaction; 2) une priorisation de la mesure d’indicateurs auprès de l’équipe proximale et 3) une prise en compte des intérêts de l’audit-feedback pour la pratique des infirmières. La démarche se conclut par une réflexion théorique sur les enjeux sous-jacents à l’aspect collectif de l’action chez les infirmières et des intérêts pour leur pratique en levier aux changements. À partir de notions de la théorie de l’Agir communicationnel de Jürgen Habermas, la pratique infirmière est abordée sous l’angle de la coordination communicationnelle et fonctionnelle. Cette problématisation théorique permet, en discussion finale de la thèse, de définir les principales pistes d’action pour concevoir et mettre en œuvre des interventions d’audit-feedback qui répondent à la fois aux intérêts des infirmières et du système de santé. / This manuscript-style thesis aims to examine from a practical and theoretical perspective the issues underlying nurses' ability to act on their practice and its context through audit and feedback interventions. Audit and feedback is commonly used in the health care to support change and improvement in professional practice. Although nurses are regularly involved in these interventions, they are studied primarily with physicians, both empirically and theoretically. The practice context of nurses has characteristics that are not conducive to the effectiveness of audit and feedback. Alvesson and Kärreman's (2007, 2011) theory development methodology was used to understand how these interventions could enable nurses to act on their practice and its context. The first component of the thesis involves an analysis of two sources of empirical data. First, a mixed-methods systematic review was conducted to understand the measured and perceived effects of audit and feedback with nurses. The data from 13 quantitative studies suggest a highly variable effectiveness of audit and feedback. The characteristics of most of the interventions were poorly aligned with current recommendations. Overall, the data from the 18 qualitative studies suggest that nurses perceive several negative aspects to audit and feedback, while recognizing the relevance of secondary use of data to support the improvement of care. The second source of data comes from a pilot study on the design, testing, and evaluation of an audit and feedback intervention with a primary care nursing team. Wound care indicators measured from the I-CLSC information system were reported to the team in two feedback sessions. Logistic regression analyses were performed over a 24-month period on 1605 wound care episodes to assess changes in indicators before, during, and after the intervention period. Only one indicator, self-management support, showed consistent improvement. Analysis of this empirical data based on an audit and feedback specific theory (Brown et al., 2019) led to three hypotheses about nurses' response to audit and feedback in the second component of the thesis. The hypotheses involve 1) integrating the relational aspect of nursing into the feedback; 2) prioritizing the use of indicators targeting teams; and 3) considering the benefits of audit and feedback for nursing practice. The theory development approach concludes with a theoretical reflection on the issues underlying the collective aspect of action among nurses and the interests for their practice as a lever for change. Using notions from Jürgen Habermas' theory of communicative action, nursing practice is viewed from the perspective of communicative and functional coordination. This theoretical perspective allows, in the final discussion of the thesis, to define the main avenues for designing and implementing audit and feedback interventions that meet the interests of both nurses and the health care system.