To ensure medical residents are prepared to work in complex and evolving settings, postgraduate medical education is transitioning to competency-based medical education, which is known as Competence by Design (CBD) in Canada. To understand how CBD is operationalized within specific residency programs and how it contributes to patient, faculty, and learner outcomes, there is a need to engage in program evaluation. However, the actual extent that, reasons for, and methods in which CBD programs are engaging in program evaluation remain unclear. Furthermore, minimal attention has been given to building program evaluation capacity within medical education programs (i.e., doing evaluation and using evaluation findings). In this research project, I explore and formally document: (a) the extent that and the ways in which CBD programs are engaging in program evaluation, (b) the reasons why these programs are engaging or not engaging in program evaluation, (c) the actual and potential positive and negative consequences of these programs engaging in program evaluation, (d) the ways that these programs build their capacities to do program evaluation and use evaluation findings, (e) the ways that program evaluators currently support these programs, and (f) the ways that program evaluators can help stakeholders build their capacities to do program evaluation and use evaluation findings. Through this research, I contribute to the limited body of empirical research on program evaluation in medical education. Confirming how CBD programs are engaging in program evaluation can advise stakeholders and program evaluators on how best to support CBD programs in building their capacities to do program evaluation and use evaluation findings, inform the design and implementation of other medical education programs, and, ultimately, enlighten program evaluation research on authentic and current evaluation practices in medical education. To meet the objectives of this study, I used a three-phase, sequential mixed methods approach. In Phase 1, I conducted a survey of Canadian program directors whose programs have transitioned to CBD to determine: (a) the extent to which CBD programs are engaging in program evaluation, and (b) the reasons why CBD programs are engaging or not engaging in program evaluation. In Phase 2, I interviewed interested program directors to explore: (c) how CBD programs are engaging in program evaluation, and (d) the ways in which CBD programs can build their capacities to do program evaluation and use evaluation findings. In Phase 3, I interviewed Canadian program evaluators to investigate: (e) how program evaluators are currently supporting CBD programs in program evaluation, and (f) how program evaluators can help CBD programs build their capacities to do program evaluation and use evaluation findings. Overall, the Phase 1 findings show that: (a) over three quarters of respondents indicated that their program does engage in program evaluation and most invite stakeholders to participate. However, most programs rarely leverage the expertise of a program evaluator and acknowledge interpreting quantitative program evaluation data is a challenge. Additionally, (b) most programs engage in program evaluation to improve their program and make decisions. However, most programs do not have an employee whose primary responsibility is program evaluation. They do not receive funding for program evaluation which affects their abilities to engage in program evaluation. Moreover, some programs do not engage in program evaluation because they do not know how to do program evaluation. The Phase 2 findings show that: (c) when program directors do engage in program evaluation, they are using ad hoc evaluation methods and a team-based format. However, program directors of CBD programs are struggling to engage in program evaluation because of limited available resources (i.e., time, financial, human resources, and technology infrastructure) and buy-in. Additionally, (d) program directors are building their capacity to do evaluation and use the findings from their specialty/subspecialty program evaluation. The Phase 3 findings show that: (e) program evaluators are supporting CBD programs by responding in a reactive way as temporary and external evaluation consultants. Finally, (f) program evaluators can help CBD programs build their capacities to do program evaluation and use the findings by using a participatory evaluation approach, leveraging existing data, encouraging the use of program evaluation approaches that are appropriate to the CBD implementation context, or encouraging programs to share findings which establishes an accountability cycle. In light of these findings, I discuss ways to engage in program evaluation, build capacity to do evaluation, and build capacity to use evaluation findings in CBD programs.

The launch of Competence by Design (CBD) in 2017 by the Royal College of Physicians and Surgeons of Canada (RCPSC) ushered in a new era of residency education in Canada. For CBD to succeed, faculty responsible for training residents must undergo faculty development to attain the relevant knowledge and skills required to fulfil their new duties. This thesis examines the faculty development resources available to faculty, and the approaches taken by program directors to facilitate faculty development. This research was guided by two research questions: (1) What faculty development resources (e.g., online modules, websites, slide decks) are currently available for faculty members in CBD programs across Canada? (2) How do program directors facilitate faculty development within their specific program? Phase 1 of this study involved a document review of all English-speaking medical schools in Canada with a post graduate CBD program. In phase 2, semi-structured interviews were conducted with program directors from the emergency medicine and psychiatry specialties. The document review found that Canadian universities hosted a range of informative websites, documents, newsletters, live sessions, and online modules to support faculty development efforts on a range of topics. During the interview phase, program directors identified live faculty development sessions, both in-person and online, were the most effective. They also expanded on their experience in the transition to CBD, some noting that their previous assessment models shared similarities with CBD, lessening the burden on faculty to change their teaching practice. Many expressed concerns over resource and time constraints on faculty development and the implementation of CBD as a whole.

Introduction: Competency-based medical education (CBME) is a learner-centered outcomes-based approach. Competence by Design (CBD) is a hybrid time-based and outcomes-based CBME model that was adopted by all Royal College of Physicians and Surgeons of Canada-based residency training programs, with the primary objective of enhancing postgraduate medical education quality. However, preliminary findings suggest that residents experience higher levels of stress, anxiety, and exhaustion in CBD than with previous curricula. This thesis aims to identify and understand the emotional effects of CBME on residents, faculty, and administrative staff. Methods: This study used a qualitative approach, specifically hermeneutic phenomenology. Seven residents, six faculty members (several with education leadership roles), and one administrative staff member from a postgraduate Psychiatry program were recruited. Participants underwent semi-structured, one-on-one interviews where they were probed on their emotions with CBME. Interviews were transcribed and analyzed using a line-by-line approach that generated individual meaning units and, subsequently, themes. Results: Five themes were identified: 1) Education is an emotional experience; 2) The emotional toll of CBD; 3) CBD is a failed educational promise—Expectations vs. realities; 4) Structural and administrative burdens of CBD; and 5) Survival of educational demands—The quest for coping. Participants initially struggled to articulate their emotions, but expressed surprise at realizing they did have strong, often negative, emotions related to CBD. There was also a dissonance identified between the anticipated benefits and the execution of CBD. Furthermore, participants highlighted administrative and structural challenges of CBD, specifically regarding Entrustable Professional Activities, which were a burden and lacked much educational value. Participants discussed using various coping strategies to manage CBD’s demands. Conclusion: The findings of this work suggest that CBD has a negative emotional impact on residents and faculty, specifically due to tension between CBD’s theoretical benefits and its practical challenges, including increased emotional burden and structural challenges. / Thesis / Master of Science (MSc) / Competency-based medical education (CBME) is a learner-centered outcomes-based approach. Competence by Design (CBD) is a hybrid time-based and outcomes-based CBME model that was adopted by all Royal College of Physicians and Surgeons of Canada-based residency training programs, with the primary objective of enhancing the quality of postgraduate medical education. However, preliminary findings suggest that residents experience higher levels of stress, anxiety, and exhaustion in a CBD model than with previous curricula. This thesis aims to understand the emotional effects of CBME on residents, faculty, and administrative staff in a postgraduate Psychiatry program. In this qualitative study, participants underwent semi-structured, one-on-one interviews where they were probed on their emotions and experiences with CBME. The findings suggest that CBD has a negative emotional impact on residents and faculty, specifically due to tension between CBD’s theoretical benefits and its practical challenges, including increased emotional burden and structural challenges associated with the assessment methods.