A validity study can be used to investigate the effectiveness of an exam and reveal both its strengths and weaknesses. This study concerns an investigation of the writing portfolio Level Achievement Test (LAT) at the English Language Center (ELC) of Brigham Young University (BYU). The writing portfolios of 251 students at five proficiency levels were rated by 11 raters. Writing portfolios consisted of two coursework essays, a self-reflection assignment, and a 30-minute timed essay. Quantitative methods included an analysis with Many-Facet Rasch Model (MFRM) software, called FACETS, which looked for anomalies in levels, classes, examinees, raters, writing criteria, and the rating scale categories. Qualitative methods involved a rater survey, rater Think Aloud Protocols (TAPs), and rater interviews. Results indicated that the exam has a high degree of validity based on the MFRM analysis. The survey and TAPs revealed that although raters follow a similar pattern for rating portfolios, they differed both in the time they took to rate portfolios and in the degree to which they favored the rating criteria. This may explain some of the discrepancies in the MFRM rater analysis. Conclusions from the MFRM analysis, surveys, TAPs, and interviews were all used to make recommendations to improve the rating process of the LAT, as well as to strengthen the relationship between LAT rating and classroom teaching and grading.

A principal argument against the use of elicited imitation (EI) to measure L2 oral proficiency is that performance does not require linguistic knowledge, but requires only rote memorization. This study addressed the issue by administering two tests to the same group of students studying English as a second language: (1) a working memory test, and (2) an English oral proficiency EI test. Participants came from a range of English language proficiency levels. A Pearson correlation was performed on the test results for each participant. The hypothesis was that English EI scores and working memory scores would not correlate significantly. This would suggest that the two tests do differ in what they measure, and that the English EI test does measure knowledge of the language to some degree. The results of the Pearson correlation revealed that there was a small positive correlation between working memory and English EI scores, but that it was not significant. There was also a significantly positive correlation between students' English EI scores and ELC level. These findings suggest that the English EI test fundamentally functions as a language test, and not significantly as a working memory test.

Speaking assessments for second language learners have traditionally been expensive to administer because of the cost of rating the speech samples. To reduce the cost, many researchers are investigating the potential of using automatic speech recognition (ASR) as a means to score examinee responses to open-ended prompts. This study examined the potential of using ASR timing fluency features to predict speech ratings and the effect of prompt difficulty in that process. A speaking test with ten prompts representing five different intended difficulty levels was administered to 201 subjects. The speech samples obtained were then (a) rated by human raters holistically, (b) rated by human raters analytically at the item level, and (c) scored automatically using PRAAT to calculate ten different ASR timing fluency features. The ratings and scores of the speech samples were analyzed with Rasch measurement to evaluate the functionality of the scales and the separation reliability of the examinees, raters, and items. There were three ASR timed fluency features that best predicted human speaking ratings: speech rate, mean syllables per run, and number of silent pauses. However, only 31% of the score variance was predicted by these features. The significance in this finding is that those fluency features alone likely provide insufficient information to predict human rated speaking ability accurately. Furthermore, neither the item difficulties calculated by the ASR nor those rated analytically by the human raters aligned with the intended item difficulty levels. The misalignment of the human raters with the intended difficulties led to a further analysis that found that it was problematic for raters to use a holistic scale at the item level. However, modifying the holistic scale to a scale that examined if the response to the prompt was at-level resulted in a significant correlation (r = .98, p < .01) between the item difficulties calculated analytically by the human raters and the intended difficulties. This result supports the hypothesis that item prompts are important when it comes to obtaining quality speech samples. As test developers seek to use ASR to score speaking assessments, caution is warranted to ensure that score differences are due to examinee ability and not the prompt composition of the test.

This thesis discusses the ability to maintain a consistent design, simulation, and test verification environment by use of the Process Model Graph (PMG) throughout the development process. This ability extends the functionality of the PMG to include the visualization of simulation results and the verification of test paths within the simulation. These ideas have been implemented within a development tool called the Modeler's Assistant. The integration of the test generation environment into the tool is discussed. The design methodology used in creating the simulation environment is also discussed. Other enhancements to increase the abilities of the tool and improve its usefulness to behavioral test generation and verification are also discussed. Many examples of the new extentions to the tool are presented. / Master of Science

Titre de l'écran-titre (visionné le 15 novembre 2023) / Cette étude a examiné l'interactivité des tâches d'écriture académique de l'IELTS et leur effet sur les pratiques de préparation des enseignants en anglais langue étrangère. En adoptant une méthodologie de recherche mixte convergente parallèle, cette étude s'est penchée sur la réalité de l'écriture académique dans des contextes réels et a examiné le degré de cohérence entre le test, les exigences d'utilisation de la langue cible et le contenu de la préparation au test. Elle a également examiné le facteur enseignant pour explorer la direction et l'intensité de l'effet de rebond de l'IELTS sur le choix du matériel, des activités en classe et de la méthodologie d'enseignement dans leurs cours de préparation au test. Dix-huit enseignants et 143 étudiants des cours de préparation à l'IELTS dans quatre écoles de langues ont participé à cette étude. La collecte de données qualitatives pour cette étude comprenait une analyse des caractéristiques des tâches effectuées sur des échantillons de tâches d'écriture académique de l'IELTS, des tâches d'écriture académique représentatives dans le contexte d'utilisation cible et des tâches en classe ; des observations en classe et des entretiens, sous forme de groupes de discussion, avec les enseignants. La collecte de données quantitatives comprenait des données provenant de deux questionnaires enseignants et de deux questionnaires étudiants. Les données ont été analysées en deux phases et triangulées afin de répondre aux questions de recherche. Dans l'ensemble, il a été constaté qu'en dépit des caractéristiques communes entre l'écriture de l'IELTS et l'écriture académique de la vie réelle, il existe des divergences majeures entre les compétences requises dans la vie réelle et les tâches du test. Il a été avancé qu'étant donné l'éventail de la diversité démographique des candidats et de l'applicabilité étendue de l'IELTS, la prise en compte adéquate de toutes les compétences requises dans la vie réelle représenterait un défi en termes de pratique pour la conception de l'IELTS. Les résultats ont mis en évidence des différences notables entre l'IELTS et l'écriture universitaire à différents niveaux en ce qui concerne l'interactivité, comme le degré d'engagement métacognitif prévu par l'interdépendance des tâches, l'engagement affectif et l'utilisation de stratégies. En ce qui concerne l'effet de rebond, l'interprétation a permis d'établir un lien entre certaines caractéristiques de la conception du test et les pratiques de préparation des enseignants en matière de préparation à l'examen, soulignant la réduction du programme d'études en raison des pressions exercées par le test, ainsi que les effets bénéfiques de la préparation à l'examen IELTS sur le comportement des enseignants, ce qui profiterait en fin de compte aux apprenants. Des facteurs tels que l'opinion des enseignants sur l'examen, leur philosophie d'enseignement, les pressions exercées par l'administration et les attentes des étudiants, entre autres, ont entraîné une variabilité de la rétroaction, influençant l'intensité et la direction des effets de l'examen sur les pratiques d'enseignement. Les retombées de l'étude comprennent l'importance des preuves de recherche et de la littératie de l'évaluation pour définir des attentes réalistes et des pratiques équitables dans les tests de langue standardisés, en particulier pour l'admission des étudiants internationaux dans les universités. L'étude suggère l'adoption d'un cadre global de littératie académique et la prise en compte de multiples critères, tels que les entretiens, les déclarations personnelles et les échantillons de textes, pour l'évaluation des candidats. Elle souligne également la nécessité pour les enseignants et les autres parties prenantes d'acquérir une littératie de l'évaluation afin de comprendre les limites et l'impact des examens à enjeux élevés. Les implications explorent en outre le potentiel des possibilités de développement professionnel, des communautés d'apprentissage en collaboration et de l'accès à des ressources de haute qualité pour améliorer les connaissances en matière d'évaluation en langue. Les résultats soulignent également le rôle des enseignants dans l'orientation des étudiants et l'influence des effets positifs et négatifs des examens à enjeux élevés. / This study investigated the interactiveness of IELTS academic writing tasks and their washback on EFL teachers' test preparation practices. Adopting a convergent parallel mixed-methods research design, this study looked into the reality of academic writing in real-life contexts and examined the degree of consistency between the test, target language use requirements, and test preparation content. It further examined the teacher factor to explore the direction and intensity of IELTS washback effect on teachers' choice of materials, class activities, and teaching methodology in their test preparation courses. Eighteen teachers and 143 students from IELTS preparation courses in four language schools participated in this study. Qualitative data collection for this study involved an analysis of task characteristics conducted on sample IELTS academic writing tasks, representative academic writing tasks in the target context of use, and classroom tasks; classroom observations, and focus-group teacher interviews. Quantitative data collection included data from two teacher questionnaires and two student questionnaires. Data was analyzed in two phases and triangulated in order to respond to the research questions. Overall, it was seen that despite shared features between IELTS writing and real-life academic writing, there are major divergences between Target Language Use (TLU) and test tasks. It was argued that given the extensive scope of test-taker demographic and wide applicability of IELTS, adequate engagement of all TLU features would pose a practicality challenge for the design of IELTS. Findings pointed to prominent differences between IELTS and university writing at various levels as far as interactiveness was concerned, such as the degree of intended metacognitive engagement via task interdependence, affective engagement, and strategy use. With regards to washback, the interpretation linked some test design features to the teachers' exam preparation practices, highlighting the narrowing of the curriculum due to the pressures imposed by the exam, as well as beneficial outcomes of IELTS exam preparation on the teachers' behavior, which would ultimately benefit the learners. Factors such as teachers' opinions of the exam, their teaching philosophy, pressures from administration and student expectations, among others, caused washback variability, influencing the intensity and direction of exam effects on teaching practices. The implications of the study include the importance of research evidence and assessment literacy in shaping realistic expectations and fair practices in standardized language tests, particularly for admitting international students to universities. The study suggests adopting a comprehensive framework of academic literacy and considering multiple criteria, such as interviews, personal statements, and writing samples, for assessing applicants. It also highlights the need for assessment literacy among teachers and other stakeholders to understand the limitations and impacts of high-stakes exams. The implications further explore the potential for professional development opportunities, collaborative learning communities, and access to high-quality resources to enhance language assessment literacy. The findings also emphasize the role of teachers in guiding students and influencing the positive and negative effects of high-stakes exams.

In neuro surgery, the expression awake surgery generally refers to tumour surgery in the brain while the patient is concious. At Akademiska Sjukhuset, Uppsala, it is today standard to perform awake surgery on tumours located in the dominant left hemisphere while awake. However, regarding awake surgery in the non-dominant right hemisphere it is not clear what functions should be tested and how this should be done. A test battery for awake surgery in the right hemisphere should be developed like the test that are used in awake surgery in the left hemisphere. Science recommends awake surgery in the right hemisphere which already being done around Europe. As patients with brain tumours have a good prognosis post operative it is important to retain, or ameliorate, their quality of life. This literature study’s main purpose is to present which functions, important for language and communication, are located in the right hemisphere. The study also intends to account for testing of these functions intraoperatively, after rightsided stroke and on persons with right hemisphere lesions. The functions are many in number, essential for maintaining the quality of life but some are difficult to test. According to the literature study, the functions that appeared most significant are neglect and pragmatic with its functional, social and communicative effects. More research on the right hemispherical functions and how they are organized is necessary. The work intends to be the basis for continued development of awake surgery tests in the right hemisphere in Akademiska Sjukhuset, Uppsala. / Inom neurokirurgin avser uttrycket vakenkirurgi oftast tumörkirurgi i hjärnan samtidigt som patienten är vid fullt medvetande under ingreppet. Vid Akademiska Sjukhuset i Uppsala är det standard att operera tumörer belägna i den språkligt dominanta vänstra hemisfären i vaket tillstånd medan man vid en del andra centra även förespråkar vakenkirurgi i den icke-språkligt dominanta högra hemisfären. Det är emellertid inte klarlagt vilka funktioner som i så fall bör testas och hur detta ska gå till. Ett testbatteri vid vakenkirurgi i höger hemisfär skulle behöva utvecklas likt det test som används vid tumörer i vänster hemisfär. Vakenkirurgi i höger hemisfär genomförs i dag på enstaka center i Europa. Då patienter med hjärntumörer kan ha en god prognos postoperativt är det angeläget att behålla, eller förbättra, deras livskvalitet. Denna litteraturstudies huvudsakliga syfte är att presentera vilka funktioner, viktiga för språk och kommunikation, som är lokaliserade i höger hemisfär. Studien redogör också för testning av dessa funktioner intraoperativt, efter högersidig stroke och på personer med högerhemisfäriska skador. Funktionerna är många till antalet, viktiga för att bibehålla livskvaliteten men en del även svårtestade. Enligt vår litteraturgenomgång är de funktioner som framstått som mest betydelsefulla neglekt och pragmatik med dess funktionella, sociala och kommuniktiva påverkan. Mer forskning om höger hemisfärs funktioner och hur dessa är organiserade behövs. Arbetet ämnar ligga till grund för fortsatt utveckling av vakenkirurgitest i höger hemisfär på Akademiska Sjukhuset i Uppsala.

This study applied the latest developments in language testing validation theory to derive a core body of evidence that can contribute to the validation of a large-scale, high-stakes English as a Foreign Language (EFL) testing program in Japan. The testing program consists of a set of seven level-specific tests targeting different levels of proficiency. This core aspect of the program was selected as the main focus of this study. The socio-cognitive model of language test development and validation provided a coherent framework for the collection, analysis and interpretation of evidence. Three research questions targeted core elements of a validity argument identified in the literature on the socio-cognitive model. RQ 1 investigated the criterial contextual and cognitive features of tasks at different levels of proficiency, Expert judgment and automated analysis tools were used to analyze a large bank of items administered in operational tests across multiple years. RQ 2 addressed empirical item difficulty across the seven levels of proficiency. An innovative approach to vertical scaling was used to place previously administered items from all levels onto a single Rasch-based difficulty scale. RQ 3 used multiple standard-setting methods to investigate whether the seven levels could be meaningfully related to an external proficiency framework. In addition, the study identified three subsidiary goals: firstly, toevaluate the efficacy of applying international standards of best practice to a local context: secondly, to critically evaluate the model of validation; and thirdly, to generate insights directly applicable to operational quality assurance. The study provides evidence across all three research questions to support the claim that the seven levels in the program are distinct. At the same time, the results provide insights into how to strengthen explicit task specification to improve consistency across levels. This study is the largest application of the socio-cognitive model in terms of the amount of operational data analyzed, and thus makes a significant contribution to the ongoing study of validity theory in the context of language testing. While the study demonstrates the efficacy of the socio-cognitive model selected to drive the research design, it also provides recommendations for further refining the model, with implications for the theory and practice of language testing validation.

This study investigates the effect of testing mode on lower-level Saudi Arabian test-takers’ performance and cognitive processes when taking an L2 reading test on computer compared to its paper-based counterpart from an interface design perspective. An interface was developed and implemented into the computer-based version of the L2 reading test in this study, which was administered to 102 Saudi Arabian University students for quantitative analyses and to an additional eighteen for qualitative analyses. All participants were assessed on the same L2 reading test in two modes on two separate occasions in a within-subject design. Statistical tests such as correlations, group comparisons, and item analyses were employed to investigate test-mode effect on test-takers’ performance whereas test-takers’ concurrent verbalizations were recorded when taking the reading test to investigate their cognitive processes. Strategies found in both modes were compared through their frequency of occurrence. In addition, a qualitative illustration of test-takers cognitive behavior was given to describe the processes when taking a lower-level L2 reading test. A mixed-method approach was adhered to when collecting data consisting of questionnaires think-aloud protocols, and post-experimental interviews as main data collection instruments. Results on test-takers’ performance showed that there was no significant difference between the two modes of testing on overall reading performance, however, item level analyses discovered significant differences on two of the test’s items. Further qualitative investigation into possible interface design related causes for these differences showed no identifiable relationship between test-takers’ performance and the computer-based testing mode. Results of the cognitive processes analyses showed significant differences in three out of the total number of cognitive processes employed by test-takers indicating that test-takers had more difficulties in processing text in the paper-based test than in the computer-based test. Both product and process analyses carried out further provided convincing supporting evidence for the cognitive validity, content validity, and context validity contributing to the construct validity of the computer-based test used in this study.

Made available in DSpace on 2015-03-05T18:11:57Z (GMT). No. of bitstreams: 0 Previous issue date: 20 / Nenhuma / Esta dissertação de mestrado investiga testagens de proficiência lingüística em língua de sinais que são aplicadas a pessoas ouvintes, intérpretes de língua de sinais, no início de sua vida profissional. Devido à diversidade de instrumentos, procedimentos e concepções do que deve ser avaliado em intérpretes de língua de sinais ILS, faz-se necessária uma investigação sobre a testagem de proficiência lingüística e a distinção entre proficiência tradutória e certificação profissional, bem como qual o momento adequado de suas aplicações nas diferentes etapas da formação e do exercício profissional dos ILS. O embasamento teórico consta da distinção entre proficiência e fluência lingüísticas; a evolução do conceito de proficiência lingüística; testagem lingüística; e de um panorama geral sobre tradução e interpretação de língua de sinais. A testagem lingüística de língua de sinais abordada neste estudo inclui aquelas explicitamente denominadas ‘de proficiência’ e as provas de habilitação ou de seleção que contenham / This is a dissertation on language proficiency testing as applied to hearing people, sign language interpreters, in the beginning of their professional lives. Due to the diversity of instruments, proceedings and conceptions of what has to be assessed in sign language interpreters SLI, an investigation on language proficiency testing and the distinction between translation proficiency and professional certification is needed, as well as when is the most adequate moment to apply different kind of testing in different phases of interpreters´ training and professional practice. The theoretical basis of this work includes the distinction between language proficiency and fluency, the evolution of the proficiency concept, language testing, and a general view about sign language translation and interpreting. The sign language testing that is approached in this study comprises those explicitly named as ´proficiency tests´ and professional or selection tests that comprise sign language proficiency features, even if the

Rater training is important for various human resource activities. The proposed symposium highlights contemporary research on rater training in different arenas such as performance appraisal, job interviews, and foreign language testing. New and emerging trends in rater training research and practice will also be discussed.