This case study examines the emergence of an autonomy-oriented assessment culture in Pediatric residency education in the School of Medicine at Queen’s University. Through a case study approach this research explores how an assessment system to capture residents’ performance in the clinical environment was initiated, how that process supported a shift in assessment cultural, and how assessment innovation is eclipsing departmental boundaries. The case has instrumental value in illuminating how an autonomy-oriented assessment system and culture can be cultivated in residency education. The analytic frame for the case was constructed based on research literature that identified weaknesses in assessment practices in residency education more generally. The approach was theoretical, with the intent to explain how a shift in assessment culture is emerging in Pediatrics. A longitudinal approach was adopted to expose shifts in the culture. The narrative structure distills the journey into a manageable story. Three watershed events that exemplified change were systematically selected from data collected over a three-year period and constitute the findings of this research. The central contribution this research makes is that it is possible to shift the culture of assessment within a Pediatric residency program. That shift can be understood to unfold over a prolonged period through a process of mediating both social and regulatory requirements. Beginning to shift the assessment culture in Pediatrics was achieved by: recognizing the need for change in assessment practice, re-conceptualizing and realizing that change, and engaging and empowering the community to support a shift in assessment culture. Strong leadership, widening community engagement and the Rubric Descriptor Bank supported this process. Five theoretically informed principles guided the emergence of the autonomy-oriented assessment culture in Pediatrics including: (a) conceptualizing learning as a social, active process: (b) focusing attention on residents’ multidimensional growth; (c) moving away from a high-stakes orientation to assessment based on the false dichotomy between formative and summative assessment and embracing it as a tool for supporting and monitoring growth over time and across contexts; (d) actively supporting residents’ learning strategy and assessment skill development; and (e) fostering a growth orientation to learning, embracing the concept of graduated autonomy. / Thesis (Ph.D, Education) -- Queen's University, 2012-12-19 21:04:59.344

Purpose of Study: Workplace-based assessment is an opportunity for a learner to be assessed in their community of practice by an expert rater. The challenges and biases brought into this assessment relationship are complex. A shift towards Competency Based Medical Education in post-graduate residency education has triggered consideration of how to implement feasible assessment tools for the operating room, the in-patient ward, and the outpatient clinic. Competent performance in outpatient clinic is vital to surgical practice, yet no assessment tool currently exists to assess daily performance of technical and nontechnical skills of surgery residents. This project describes the development of a competency-based assessment tool, the Ottawa Clinic Assessment Tool (OCAT). Research Question: How does the OCAT demonstrate validity for measurement of surgical resident performance in clinic? Method: A consensus group of experts was gathered to generate ideas reflective of a competent ‘generalist’ surgeon in clinic. An entrustability anchor scale was developed. A six-month pilot study of the OCAT was conducted in orthopedics, general surgery and obstetrics and gynecology with quantitative and qualitative evidence of validity collected. Two subsequent feedback sessions, and a survey for staff and residents evaluated the OCAT for clarity and utility. Results: The OCAT was developed as a 13-item tool, with a global assessment item and 2 short answers questions. 44 staff surgeons completed 132 OCAT assessments of 79 residents. Psychometric data was collected as evidence of internal structure validity and relations with other variables. Analysis of feedback indicated the rating scale was practical and useful for surgeons and residents. Conclusions & Contribution to the Research Field: Surgical programs will require a daily clinic assessment tool to help define resident competency progression. Multiple sources of validity evidence collected in this pilot project demonstrate that the OCAT can measure resident clinic competency in a valid and feasible manner.

Purpose: Resilience, the capacity to endure and overcome hardship, has been suggested as a basic competency for rural medical practice. Unfortunately for physician educators, the medical education literature offers only limited guidance for nurturing this adaptive capacity. We describe the process and subsequent analysis of a daylong curriculum development workshop conducted at the annual meeting of Rural Medical Educators in 2010. Methods: Fifty administrator, faculty and student attendees reflected individually and worked in groups to construct key curricular components and modalities for teaching this competency. Prior to the meeting, participants were asked to submit a personal story about resilience. The 22 narratives received were distributed across 8 groups and provided the grist for the small group discussions, in which each group identified key concepts for teaching and learning about resilience, constructed a concept map, and developed a curriculum that was presented to all session participants. Concept maps, curriculum outlines and notes taken during the presentations were analyzed using content analysis techniques. Findings: Data highlight the importance of (1) embracing hardship as an opportunity for growth, (2) viewing resilience as both an individual and community property, (3) pursuing adaptability more than hardiness, and (4) setting a lifelong pattern of learning this competency in practice. Specific teaching modalities are suggested including individual reflective time and group activities. Conclusions: To our knowledge this represents a first effort to define and develop a medical curriculum for teaching resiliency in rural predoctoral and residency education.

Background: There is considerable evidence that physicians lack sufficient training in facilitating transition from pediatric to adult care systems for adolescents and young adults (A/YA). While several primary care residency programs have introduced health care transition (HCT) curricula in recent years, there are few studies that assess the effectiveness of HCT teaching models. Purpose: To assess the impact of a residency education program that uses electronic health records (EHR) and other methods to teach residents how to prepare A/YA for transition to adult care. Methods: In a mixed methods, quasi-experimental research design, quantitative methods were used to measure change in knowledge, confidence and experience among 67 Pediatrics and Med-Peds residents who participated in the program. All residents and a comparison group were invited to complete a 35-item pre/post-survey; a retrospective chart review provided documentation of age-specific HCT preparation tasks completed by residents during well visits for A/YA aged 12-21. Descriptive and correlational analyses were conducted to compare differences between resident and control test scores for 5 outcome variables, and to measure resident utilization of the HCT tool in the EHR. Using the Reach Effectiveness-Adoption Implementation Maintenance (RE-AIM) evaluation model as a guide, semi-structured interviews were conducted concurrently with residents and faculty to assess program acceptability, feasibility, and other important attributes. Interviews were transcribed and analyzed using a constant comparative, iterative process. Results: Survey results showed residents (11 matched pairs) scored significantly higher than controls (13 matched pairs) in 2 of 5 outcomes: exposure to HCT learning activities (p=.0005) and confidence in providing primary care for YSHCN (p=.0377). Overall utilization of the EHR tool among 51 residents was 52.8% (57 of 108 patient visits). In interviews conducted with 16 residents and 6 faculty, both groups said that HCT training is a highly relevant need. Residents said they had little knowledge or experience in HCT prior to the intervention but felt more confident in their abilities afterwards. The HCT tool in the EHR was the only intervention element among multiple modalities that reached all study participants, with more than 80% of residents interviewed reporting they used the HCT tool "usually" or "always." Factors that influenced program adoption included accessibility of educational materials, ease of use, time constraints, patient age and health condition, and attending physicians' enforcement of the protocol. Conclusion: This study contributes to the body of knowledge concerning HCT by increasing our understanding of ways to effectively educate residents about transition preparation. Results show a positive intervention effect on selected dimensions of resident knowledge, confidence, and practice in HCT, highlighting program strengths and weaknesses. The program is distinctive in educating residents to prepare all A/YA for HCT, as recommended by major medical associations for pediatric and adult care physicians, and in its use of the EHR as a primary teaching tool, a consideration for reducing time-intensive didactic instruction. It provides a model that can be adapted by other residency and provider training programs, and suggests a need to integrate acquisition of health care self-management skills more broadly in child and adolescent health preventive care tools and policies.