No description available.

Prior studies suggest that raters' familiarity with test-takers' first language (L1) can be a potential source of bias in rating speaking tests. However, there is still no consensus between researchers on how and to what extent that familiarity affects the scores. This study investigates raters' performance and focuses on not only how raters' second language (L2) proficiency level interacts with examinees' L1, but also if raters' teaching experience has any effect on the scores. Speaking samples of 58 ESL learners with L1s of Spanish (n = 30) and three Asian languages (Korean, n = 12; Chinese, n = 8; and Japanese, n = 8) of different levels of proficiency were rated by 16 trained raters with varying levels of Spanish proficiency (Novice to Advanced) and different degrees of teaching experience (between one and over 10 semesters). The ratings were analyzed using Many-Facet Rasch Measurement (MFRM). The results suggest that extensive rater training can be quite effective: there was no significant effect of either raters' familiarity with examinees' L1, or raters' teaching experience on the scores. However, even after training, the raters still exhibited different degrees of leniency/severity. Therefore, the main conclusion of this study is that even trained raters may consistently rate differently. The recommendation is to (a) have further rater training and calibration; and/or (b) use MFRM with fair average to compensate for the variance.

This dissertation centers on the under-researched construct of interactional competence, which refers to features of jointly constructed discourse. When applied to the testing of speaking skills in a second language, interactional competence refers to features of the discourse that the two students produce together; rather than the speaking ability or performance of each person individually. This dissertation describes the construct of interactional competence in a low-stakes, paired speaking test setting targeted at students in their second year of German instruction at the college level. The purpose of this study is two-fold. First, the study analyzes the conversational resources that are co-constructed in the test discourse to maintain mutual understanding, which is considered the basis for interactional competence. Second, the study examines the impact of task (jigsaw task and discussion task) and speaking ability-level combination (same and different ability) in the test-taker pair on the co-constructed test discourse and thus on the deployment of the conversational resources to maintain intersubjectivity. In that respect, this study also seeks to analyze how the identified conversational resources are involved in establishing and negotiating language ability identities that are displayed in the test discourse. Conversation analytic conventions were used to investigate the interactional resources that test takers deploy to maintain mutual understanding. The procedures of repair (self-repair in response to other-initiated repair, inter-turn delays, and misunderstandings as well as other-repair in conjunction with word search activities) that emerged from the inductive analysis of the test discourse have broadened the conceptualization of interactional competence in the context of paired speaking assessments. Frequency distributions of the interactional resources were created to provide a better understanding of the impact of task and ability-level combination on the co-constructed repair procedures. The rationale behind this analysis is the general understanding of language testers that both resources and context influence test performance. The findings from the quantitative analysis suggest that there are more similarities than differences in repair use across the jigsaw task and the discussion task. In addition, even though some trends in the co-construction of repair procedures may be attributed to the higher or lower speaking ability of the test takers, the relationship between the ability-level combination in the pair and the use of repair seems to be rather variable. Finally, to learn more about the interrelationship between test takers’ speaking ability and interactional competence, this dissertation also approached speaking ability in terms of test takers’ co-constructed language ability identities that are displayed in the test discourse. By means of single case analyses, the study provided a detailed picture of the relationship between language ability identities and the procedures of repair, both of which are co-constructed at the discourse level. The findings from the conversation analysis show that the speaker who provides the repair is usually able to position himself or herself as the more competent or proficient speaker in the test discourse.

This thesis explores relations between learner culture, operationalized as degree of individualism/collectivism (I/C), and English-as-an-additional-language (EAL) speaking test performance with two test formats that differ in terms of interactiveness. Seven Korean participants’ speaking test performances with the two different formats were compared. Results did not differentiate the speaking test formats in terms of mean speaking test scores or gains. However, results supported the value of the interactive format – Dynamic Assessment (DA) – for discriminating between test-takers in terms of grammatical and lexical performance. This characteristic suggests DA’s potential effectiveness as a component of a formal speaking test, particularly for ongoing classroom testing and/or exit testing. I/C scores did not correlate significantly with scores on the two speaking test formats. However, qualitative analysis based on I/C scores identified differences in the ways that participants oriented themselves towards accuracy or task topics in corrective exchanges during DA tests. Participants’ email survey responses supported this analysis. These findings are commiserate with reports of accuracy focus in Korean educational culture. This link points to the value of future I/C research focusing on accuracy/task-focus orientations. To more reliably demonstrate relations between I/C and EAL performance, this study’s discussion describes a more direct I/C measurement approach.

The present study examines possible effects of extramural English (EE) on oral proficiency (OP) and vocabulary (VOC). The study is based on data collected from Swedish learners of ESL in grade 9 (aged 15-16; N=80; 36 boys, 44 girls) over a period of one year. EE was defined as linguistic activities that learners engage in outside the classroom in their spare time. EE was measured with the help of a questionnaire and two language diaries, each covering one week. In the diaries, the learners recorded how much time they had spent on seven given EE activities (reading books, reading newspapers/magazines, watch­ing TV, watching films, surfing the Internet, playing video games, listening to music). There was also an open category. Speech data were collected with the help of five interactional speaking tests; learners were in random dyads on each occasion. Each student performance was assessed by three raters with the help of a profile scheme, resulting in an overall grade. Based on these grades from the tests, a mean grade for OP (the OP grade) was calculated for each student. OP was defined as the learner’s ability to speak and use the target language in actual communication with an interlocutor. Learners’ VOC was measured with an index variable based on the scores on two written vocabulary tests. For a selection of ten learners, additional analyses were made of oral fluency and the use of advanced vocabulary in speech. A mixed methods research design was used, but the lion’s share of data was analyzed using inferential statistics. Results showed that the total amount of time spent on EE correlated positively and significantly (p < .01) both with learners’ level of OP and size of VOC, but that the correlation between EE and VOC was stronger and more straightforward than the one between EE and OP. The conclusion drawn was that although EE impacts both OP and VOC, the causal relationship is more salient in the case of VOC. Results also showed that some activities were more important than others for OP and VOC respectively; i.e., the type of EE activity mattered. EE activities that required learners to be more productive and rely on their language skills (video games, the Internet, reading) had a greater impact on OP and VOC than activities where learners could remain fairly passive (music, TV, films). An important gender difference was identified. Boys spent significantly more time on productive EE activities than girls; therefore, EE had a greater impact on OP and VOC for boys than for girls. Four background variables were also studied. The conclusion was that EE is an independent variable and a possible path to progress in English for any learner, regard­less of his or her socioeconomic background.