Purpose of Study: Workplace-based assessment is an opportunity for a learner to be assessed in their community of practice by an expert rater. The challenges and biases brought into this assessment relationship are complex. A shift towards Competency Based Medical Education in post-graduate residency education has triggered consideration of how to implement feasible assessment tools for the operating room, the in-patient ward, and the outpatient clinic. Competent performance in outpatient clinic is vital to surgical practice, yet no assessment tool currently exists to assess daily performance of technical and nontechnical skills of surgery residents. This project describes the development of a competency-based assessment tool, the Ottawa Clinic Assessment Tool (OCAT). Research Question: How does the OCAT demonstrate validity for measurement of surgical resident performance in clinic? Method: A consensus group of experts was gathered to generate ideas reflective of a competent ‘generalist’ surgeon in clinic. An entrustability anchor scale was developed. A six-month pilot study of the OCAT was conducted in orthopedics, general surgery and obstetrics and gynecology with quantitative and qualitative evidence of validity collected. Two subsequent feedback sessions, and a survey for staff and residents evaluated the OCAT for clarity and utility. Results: The OCAT was developed as a 13-item tool, with a global assessment item and 2 short answers questions. 44 staff surgeons completed 132 OCAT assessments of 79 residents. Psychometric data was collected as evidence of internal structure validity and relations with other variables. Analysis of feedback indicated the rating scale was practical and useful for surgeons and residents. Conclusions & Contribution to the Research Field: Surgical programs will require a daily clinic assessment tool to help define resident competency progression. Multiple sources of validity evidence collected in this pilot project demonstrate that the OCAT can measure resident clinic competency in a valid and feasible manner.

This thesis investigates problems with current assessment and feedback practices in surgical training, and proposes ways to mediate some of these issues. In the first study, we report a number of barriers that might prevent supervisors from providing constructive feedback to medical trainees, including a fear of being labeled as intimidating or harassing. We also identify some barriers which make it challenging for supervisors to accurately report the progress of underperforming trainees including insufficient documentation, a perceived lack of support from program leadership, insufficient opportunities to observe performance, competing demands on time, fear of appeals or legal action, and fear of reciprocated poor staff evaluations. The second study examines the effectiveness of two new tools for assessing trainees’ performance and determines that these tools might be helpful in overcoming some of the barriers identified in the first study to help supervisors provide more meaningful feedback to trainees. Participants reported that the new tools were helpful for improving learning in six ways: by sparking a dialogue regarding performance; by promoting a structured, procedure-specific framework which could be used as a mechanism for providing high quality feedback; by providing a framework for directing future learning; by facilitating a systematic tracking of progress over time; by promoting an increased motivation to learn; and by introducing a learning focus to the intraoperative environment. Together, the data from this thesis provide us with a better understanding of how we might improve the use of assessments and feedback in surgical training. The hope is that our findings will improve the education of new physicians and ultimately improve patient care. / Thesis / Master of Science (MSc) / This thesis investigates problems with current assessment and feedback practices in surgical training, and proposes ways to mediate some of these issues. We present two studies. In the first, we explore barriers that might prevent supervisors from providing constructive feedback to medical trainees. We also identify some barriers which make it challenging for supervisors to accurately report the progress of underperforming trainees. The second study examines the effectiveness of two new tools for assessing trainees’ performance and demonstrates that these tools might be helpful in overcoming some of the barriers identified in the first study to help supervisors provide more meaningful feedback to trainees. Together, the data from this thesis provide us with a better understanding of how we might improve the use of assessment and feedback in surgical training. The hope is that our findings will improve the education of new physicians and ultimately improve patient care.

The launch of Competence by Design (CBD) in 2017 by the Royal College of Physicians and Surgeons of Canada (RCPSC) ushered in a new era of residency education in Canada. For CBD to succeed, faculty responsible for training residents must undergo faculty development to attain the relevant knowledge and skills required to fulfil their new duties. This thesis examines the faculty development resources available to faculty, and the approaches taken by program directors to facilitate faculty development. This research was guided by two research questions: (1) What faculty development resources (e.g., online modules, websites, slide decks) are currently available for faculty members in CBD programs across Canada? (2) How do program directors facilitate faculty development within their specific program? Phase 1 of this study involved a document review of all English-speaking medical schools in Canada with a post graduate CBD program. In phase 2, semi-structured interviews were conducted with program directors from the emergency medicine and psychiatry specialties. The document review found that Canadian universities hosted a range of informative websites, documents, newsletters, live sessions, and online modules to support faculty development efforts on a range of topics. During the interview phase, program directors identified live faculty development sessions, both in-person and online, were the most effective. They also expanded on their experience in the transition to CBD, some noting that their previous assessment models shared similarities with CBD, lessening the burden on faculty to change their teaching practice. Many expressed concerns over resource and time constraints on faculty development and the implementation of CBD as a whole.

Competence committees are groups of educators that monitor the progress of medical trainees and decide when they should be promoted to the next stage of training. They represent an important part of modern-day competency-based medical education programs, yet relatively little is known about their implementation and decision-making practices. This thesis seeks to fill a critical gap in the literature by generating empirical evidence with respect to competence committee implementation and decision-making practices across multiple programs. The first data chapter uses a multi-method approach to examine competence committee implementation practices at a Canadian institution over a three-year period. The second and third chapters examine how individuals and groups make promotion decisions, respectively. These chapters also consider the role of non-traditional data sources, such as anecdotal evidence, in competence committees’ decision-making processes. The final data chapter considers the role of social influences and power and examines how factors such as members’ position on the committee, gender, and race/ethnicity influence their contributions to the committee. This thesis provides insight into some of the challenges that exist with respect to competence committee implementation and offers potential solutions based on best practices across multiple programs. It also highlights factors that can influence competence committee decision making and discusses ways that their decision-making processes can be optimized. Broader implications of this thesis, including the role of groups in solving complex problems and the importance of diversity (both in terms of demographics and functional specialization) in ensuring good decision-making outcomes, are also discussed. / Thesis / Doctor of Philosophy (PhD) / Competence committees are groups of experienced health professionals and educators whose job is to determine whether physician learners (i.e., residents) are ready to progress to the next stage of training and responsibility. These committees are relatively new, and as a result, we do not know very much about how they make decisions. Given the importance of competence committees in ensuring that physicians are able to provide high-quality and safe patient care, the purpose of this thesis was to examine competence committee implementation and decision-making practices at a Canadian academic centre. This took place in two parts. First, we studied competence committees over a three-year period using surveys, interviews, and observations. This helped us understand some of their benefits and challenges. Next, we conducted a series of experiments to understand how competence committee members make decisions both individually and as part of a group. These experiments also helped us understand how competence committees make sense of different types of data, such as prior knowledge about a resident or their assessors. Finally, we examined how various aspects of members’ social identities, such as their position on the committee, their gender, and their race/ethnicity, influence their contributions to the committee. Collectively, the findings of this thesis help to advance the scientific literature in the areas of medical education and group decision making. They can also be used to optimize competence committee operations, which can in turn positively impact patients, healthcare, and society.

Introduction: Competency-based medical education (CBME) is a learner-centered outcomes-based approach. Competence by Design (CBD) is a hybrid time-based and outcomes-based CBME model that was adopted by all Royal College of Physicians and Surgeons of Canada-based residency training programs, with the primary objective of enhancing postgraduate medical education quality. However, preliminary findings suggest that residents experience higher levels of stress, anxiety, and exhaustion in CBD than with previous curricula. This thesis aims to identify and understand the emotional effects of CBME on residents, faculty, and administrative staff. Methods: This study used a qualitative approach, specifically hermeneutic phenomenology. Seven residents, six faculty members (several with education leadership roles), and one administrative staff member from a postgraduate Psychiatry program were recruited. Participants underwent semi-structured, one-on-one interviews where they were probed on their emotions with CBME. Interviews were transcribed and analyzed using a line-by-line approach that generated individual meaning units and, subsequently, themes. Results: Five themes were identified: 1) Education is an emotional experience; 2) The emotional toll of CBD; 3) CBD is a failed educational promise—Expectations vs. realities; 4) Structural and administrative burdens of CBD; and 5) Survival of educational demands—The quest for coping. Participants initially struggled to articulate their emotions, but expressed surprise at realizing they did have strong, often negative, emotions related to CBD. There was also a dissonance identified between the anticipated benefits and the execution of CBD. Furthermore, participants highlighted administrative and structural challenges of CBD, specifically regarding Entrustable Professional Activities, which were a burden and lacked much educational value. Participants discussed using various coping strategies to manage CBD’s demands. Conclusion: The findings of this work suggest that CBD has a negative emotional impact on residents and faculty, specifically due to tension between CBD’s theoretical benefits and its practical challenges, including increased emotional burden and structural challenges. / Thesis / Master of Science (MSc) / Competency-based medical education (CBME) is a learner-centered outcomes-based approach. Competence by Design (CBD) is a hybrid time-based and outcomes-based CBME model that was adopted by all Royal College of Physicians and Surgeons of Canada-based residency training programs, with the primary objective of enhancing the quality of postgraduate medical education. However, preliminary findings suggest that residents experience higher levels of stress, anxiety, and exhaustion in a CBD model than with previous curricula. This thesis aims to understand the emotional effects of CBME on residents, faculty, and administrative staff in a postgraduate Psychiatry program. In this qualitative study, participants underwent semi-structured, one-on-one interviews where they were probed on their emotions and experiences with CBME. The findings suggest that CBD has a negative emotional impact on residents and faculty, specifically due to tension between CBD’s theoretical benefits and its practical challenges, including increased emotional burden and structural challenges associated with the assessment methods.