High-stakes testing is used for the purposes of providing results that have important consequences such as certifications, licensing, or credentialing. The purpose of this study was to examine aspects of an exam recently written by flight paramedics for recertification and make recommendations for development of future exams. In 2008, an unexpectedly high failure led to revisions in the exam development process for flight paramedics. Using principles of classical test theory and generalizability theory, I examined the decision consistency and dependability of the examination and found the decision consistency for dichotomous items to be within acceptable limits, yet the dependability was low. Discrimination was strong at the cut-score. An in-depth look into the process used to set the exam, as well as the psychometric properties of the exam and the items have led to recommendations that will contribute to future development of dependable exams in the industry that result in more valid interpretations with respect to paramedic competence. / Thesis (Master, Education) -- Queen's University, 2012-09-06 22:41:41.552

This dissertation presents multiple facets of the assessment system reform of the School Leaving Certificate (SLC) exam in Nepal through grounded theory methodology. The main purpose of this study is to develop a substantive grounded theory of the reform that explicates a complete process of the assessment reform, i.e., how this reform was conceptualized, what was done as a foundational preparation, how it was implemented, and what were the preliminary responses to the reform implementation. In order to theorize the reform process, this study applied a constructivist grounded theory approach, specifically based on the work of Charmaz (1996, 2005, 2006, 2012), as the research methodology. The data were collected by conducting 10 focus group discussions and 24 one-on-one semi-structured interviews. The participants represented almost all the categories of stakeholders (e.g., policymakers, employers, teachers, students) as well as experts and educators. The emergent conceptual categories and sub-categories from the interviews and focus group discussions were categorized under three constructs: (1) Conceptualization of the reform, (2) implementation of the reform and (3) reform effects. The first construct covers those theme categories and sub-categories that state the origin and type of the reform, reform aims, and framework as well as preparation for the reform. Similarly, the second construct includes those theme categories and sub-categories that inform the mechanism underpinning reform related information dissemination, the strategic plan used to implement the reform and factors that could influence the reform process. Finally, the third construct comprises those categories that discuss reform associated quandaries and condemnation, factors contributing to magnifying the quandaries, resultant opportunities from the reform, reform impact, step forward, and other relevant categories. The findings have been explicated under three phases- the pre-implementation phase, the implementation phase and the post-implementation phase as the integrated grounded theory. The theoretical components presented under the pre-implementation phase include exhaustive analysis of need and feasibility; input from experts, educators and key stakeholders; clarity on reform aims and objectives; cooperative triangular relationships; comprehensive documentation; explicit roadmap and exhaustive planning; infrastructure and resource management, and capacity building. Similarly, three major theoretical components- teacher advocacy, stakeholder ownership, and timely and authentic information have been described under the implementation phase. Finally, eight major theoretical components, such as identification and analysis of resulting issues, immediate actions for the urgent/sensitive issues, special programs for the low-grade holders, bridge between academic and vocational programs, need for impact analysis, effective communication channel, need for institutional memory, need for an unconventional assessment system, and need for a resourceful unit of assessment and testing have been presented under the post-implementation phase. The combination of the theoretical components described under the three phases mentioned above is the integrated substantive grounded theory of the assessment system reform in Nepal. This study contributes by adding value for those involved in assessment reform as well as the academicians and researchers because it puts forward recommendations for foundational preparation and homework in the conceptual phase of assessment reform; action steps to minimize the possible resistance to reform; strategies to implement the reform successfully; and initiatives to institutionalize the reform or address the resulting issues and concerns.

Taiwan English teachers in secondary schools are facing a new challenge of having to teach listening comprehension skills to their students. Many instructors are not equipped to deliver adequate listening instruction although listening comprehension has become a benchmark for senior high students aspiring to enter a tertiary institution and earn a college degree. Instructors, students, parents, administrators, and members of the Ministry of Education agree that there is a need to meet the demands of the National English Curriculum Standards. Weimar's approach of learner-centered teaching served as the conceptual foundation for this study since the focus was on ways teachers could work collaboratively to learn evidence-based strategies for teaching listening comprehension skills. Guided by 4 research questions, the study investigated the perspectives of 4 teacher-participants at a Northern District Senior High regarding the process of teaching listening comprehension. In this qualitative study, an inductive analysis was used to define themes and concepts to discern any patterns and relationships connected with the data collected: interviews, classroom observation and unobtrusive measures. The findings showed that teacher-led peer coaching, via collegial collaboration, led to the building of teaching communities and more effective use of evidence-based strategies for improving the teaching of listening comprehension. This improved teacher pedagogy may lead to a positive social change whereby teachers have skills for teaching listening comprehension and students are better prepared for further schooling. Components of the project also can be used by other school districts and educational organizations where the administration can use the model to further advance similar workforce capacity in teaching L2 learners listening instruction.