Elicited Imitation (EI), which is a way of assessing language learners' speaking, has been used for years. Furthermore, there have been many studies done showing rater bias (variance in test ratings associated with a specific rater and attributable to the attributes of a test taker) in language assessment. In this project, I evaluated possible rater bias, focusing mostly on bias attributable to raters' and test takers' language backgrounds, as seen in EI ratings. I reviewed literature on test rater bias, participated in a study of language background and rater bias, and produced recommendations for reducing bias in EI administration. Also, based on possible rater bias effects discussed in the literature I reviewed and on results of the research study I participated in, I created a registration tool to collect raters' background information that might be helpful in evaluating and reducing rater bias in future EI testing. My project also involved producing a co-authored research paper. In that paper we found no bias effect based on rater first or second language background.

No description available.

There is no comprehensive model of language test or examination impact and how it might be investigated within educational contexts by a provider of high-stakes examinations, such as an international examinations board. This thesis addresses the development of such a model from the perspective of Cambridge ESOL, a provider of English language tests and examinations in over 100 countries. The starting point for the thesis is a discussion of examinations within educational processes generally and the role that examinations board, such as Cambridge ESOL play within educational systems. The historical context and assessment tradition is an important part of this discussion. In the literature review, the effects and consequences of language tests and examinations are discussed with reference to the better known concept of washback and how impact can be defined as a broader notion operating at both micro and macro levels. This is contextualised within the assessment literature on validity theory and the application of innovation theories within educational systems. Methodologically, the research is based on a meta-analysis which is employed in order to describe and review three impact projects. These three projects were carried out by researchers based in Cambridge to implement an approach to test impact which had emerged during the 1990s as part of the test development and validation procedures adopted by Cambridge ESOL. Based on the analysis, the main outcome and contribution to knowledge is an expanded model of impact designed to provide examination providers with a more effective “theory of action”. When applied within Cambridge ESOL, this model will allow anticipated impacts of the English language examinations to be monitored more effectively and will inform on-going processes of innovation; this will lead to well-motivated improvements in the examinations and the related systems. Wider applications of the model in other assessment contexts are also suggested.

Elicited Imitation (EI) is an assessment approach that uses sentence imitation tasks to gauge the oral proficiency level of test takers. EI tests have been created for several of the world's languages, including English, Spanish, Japanese, French, and Mandarin. Little research has been conducted for using the EI approach with learners of Russian. This dissertation describes a multi-faceted study that was presented in two journal articles for the creation and analysis of a Russian EI test. The EI test was created for and tested with Russian-speaking missionaries and employees at the Missionary Training Center (MTC) in Provo, UT. The first article describes the creation of the test and analyzes its ability to predict oral language proficiency by comparing individuals' scores on the EI to their scores on the Oral Proficiency Interview (OPI). The test was found to effectively predict an individual's OPI score (R2 = .86). The second article analyzes the difference in person ability estimates and item difficulty measures between items from a general content bank and a religious content bank. The mean score for the content specific items (x̄ = .51) was significantly higher than the mean score for the general test (x̄ = .44, p < 0.001). Additionally, the item difficulties for the religious items were significantly less than the item difficulties for the general items (p < 0.05).

This study applied the latest developments in language testing validation theory to derive a core body of evidence that can contribute to the validation of a large-scale, high-stakes English as a Foreign Language (EFL) testing program in Japan. The testing program consists of a set of seven level-specific tests targeting different levels of proficiency. This core aspect of the program was selected as the main focus of this study. The socio-cognitive model of language test development and validation provided a coherent framework for the collection, analysis and interpretation of evidence. Three research questions targeted core elements of a validity argument identified in the literature on the socio-cognitive model. RQ 1 investigated the criterial contextual and cognitive features of tasks at different levels of proficiency, Expert judgment and automated analysis tools were used to analyze a large bank of items administered in operational tests across multiple years. RQ 2 addressed empirical item difficulty across the seven levels of proficiency. An innovative approach to vertical scaling was used to place previously administered items from all levels onto a single Rasch-based difficulty scale. RQ 3 used multiple standard-setting methods to investigate whether the seven levels could be meaningfully related to an external proficiency framework. In addition, the study identified three subsidiary goals: firstly, toevaluate the efficacy of applying international standards of best practice to a local context: secondly, to critically evaluate the model of validation; and thirdly, to generate insights directly applicable to operational quality assurance. The study provides evidence across all three research questions to support the claim that the seven levels in the program are distinct. At the same time, the results provide insights into how to strengthen explicit task specification to improve consistency across levels. This study is the largest application of the socio-cognitive model in terms of the amount of operational data analyzed, and thus makes a significant contribution to the ongoing study of validity theory in the context of language testing. While the study demonstrates the efficacy of the socio-cognitive model selected to drive the research design, it also provides recommendations for further refining the model, with implications for the theory and practice of language testing validation.

Syntactic complexity has been an area of significant interest in L2 writing development studies over the past 45 years. Despite the regularity in which syntactic complexity measures have been employed, the construct is still relatively under-developed, and, as a result, the cumulative results of syntactic complexity studies can appear opaque. At least three reasons exist for the current state of affairs, namely the lack of consistency and clarity by which indices of syntactic complexity have been described, the overly broad nature of the indices that have been regularly employed, and the omission of indices that focus on usage-based perspectives. This study seeks to address these three gaps through the development and validation of the Tool for the Automatic Assessment of Syntactic Sophistication and Complexity (TAASSC). TAASSC measures large and fined grained clausal and phrasal indices of syntactic complexity and usage-based frequency/contingency indices of syntactic sophistication. Using TAASSC, this study will address L2 writing development in two main ways: through the examination of syntactic development longitudinally and through the examination of human judgments of writing proficiency (e.g., expert ratings of TOEFL essays). This study will have important implications for second language acquisition, second language writing, and language assessment.

The association between receptive language skills and reading comprehension has been established in the research literature. Even when the importance of receptive skills for reading comprehension has been strongly supported, in practice lower levels of skills tend to go unnoticed in typically developing children. A potentially more visible modality of language, expressive skills using speech samples, has been rarely examined despite the longitudinal links between speech and later reading development, and the connections between language and reading impairments. Even fewer reading studies have examined expressive skills using a subgroup of speech samples – narrative samples – which are closer to the kind of language practitioners can observe in their classrooms, and are also a rich source of linguistic and discourse-level data in school-aged children. This thesis presents a study examining the relationship between expressive language skills in narrative samples and reading comprehension after the first two years of formal reading instruction, with considerable attention given to methodological and developmental issues. In order to address the main methodological issues surrounding the identification of the optimal linguistic indices in terms of reliability and the existence of developmental patterns, two studies of language development in oral narratives were carried out. The first of the narrative language studies drew data from an existing corpus, while the other analysed primary data, collected specifically for this purpose. Having identified the optimal narrative indices in two different samples, the main study examined the relationships between these expressive narrative measures along with receptive standardised measures, and reading comprehension in a monolingual sample of eighty 7- and 8-year-old children attending Year 3 in the UK. Both receptive and expressive oral language skills were assessed at three different levels: vocabulary, grammar and discourse. Regression analyses indicated that, when considering expressive narrative variables on their own, expressive grammar and vocabulary, in that order, contributed to explain over a fifth of reading comprehension variance in typically developing children. When controlling for receptive language however, expressive skills were not able to account for significant unique variance in the outcome measure. Nonetheless, mediation analyses revealed that receptive vocabulary and grammar played a mediating role in the relationship between expressive skills from narratives and reading comprehension. Results and further research directions are discussed in the context of this study’s methodological considerations.

In second language (L2) listening assessment, various factors have the potential to impact the validity of listening test items (Brindley & Slatyer, 2002; Buck & Tatsuoka, 1998; Freedle & Kostin, 1999; Nissan, DeVincenzi, & Tang, 1996; Read, 2002; Shohamy & Inbar, 1991). One relatively unexplored area to date is who controls the aural input. In traditional standardized listening tests, an administrator controlled recording is played once or twice. In real world or classroom listening, however, listeners can sometimes request repetition or clarification. Allowing listeners to control the aural input thus has the potential to add test authenticity but requires careful design of the input and expected response as well as an appropriate computer interface. However, if candidates feel less anxious, allowing control of listening input may enhance examinees' experience and still reflect their listening proficiency. Comparing traditional and self paced (i.e., examinees having the opportunity to start, stop, and move the audio position) delivery of multiple choice comprehension items, my research inquiry is whether self paced listening can be a sufficiently reliable and valid measure of examinees' listening ability. Data were gathered from 100 prospective and current university ESL students. They were administered computer based multiple choice listening tests: 10 identical once played items, followed by 33 items in three different conditions: 1) administrator paced input with no audio player visible, 2) self paced with a short time limit, and 3) self paced with a longer time limit. Many facet Rasch (1960/1980) modeling was used to compare the difficulty and discrimination of the items across conditions. Results indicated items on average were similar difficulty overall but discriminated best in self paced conditions. Furthermore, the vast majority of examinees reported they preferred self paced listening. The quantitative results were complemented by follow up stimulated recall interviews with eight participants who took 22 additional test items using screen capture software to explore whether and when they paused and/or repeated the input. Frequency of and reasons for self pacing did not follow any particular pattern by proficiency level. Examinees tended to play more than once but not two full times through, even without limited time. Implications for listening instruction and classroom assessment, as well as standardized testing, are discussed.

The current study investigates the construct representation of the reading component of a B2 level general English test: First Certificate in English (FCE). Construct representation is the relationship between cognitive processes elicited by the test and item difficulty. To facilitate this research, a model of the cognitive process involved in responding to reading test items was defined, drawing together aspects of different models (Embretson & Wetzel, 1987; Khalifa & Weir, 2009; Rouet, 2012). The resulting composite contained four components: the formation of an understanding of item requirements (OP), the location of relevant text in the reading passage (SEARCH), the retrieval of meaning from the relevant text (READ) and the selection of an option for the response (RD). Following this, contextual features predicted by theory to influence the cognitive processes, and hence the difficulty of items, were determined. Over 50 such variables were identified and mapped to each of the cognitive processes in the model. Examples are word frequency in the item stem and options for OP; word frequency in the reading passage for READ; semantic match between stem/option and relevant text in the passage for SEARCH; and dispersal of relevant information in the reading passage for RD. Response data from approximately 10,000 live test candidates were modelled using the Linear Logistic Test Model (LLTM) within a Generalised Linear Mixed Model framework (De Boeck & Wilson, 2004b). The LLTM is based on the Rasch model, for which the probability of success on an item is a function of item difficulty and candidate ability. The holds for LLTM except that item difficulty is decomposed so that the contribution of each source of difficulty (the contextual features mentioned above) is estimated. The main findings of the study included the identification of 26 contextual features which either increased or decreased item difficulty. Of these features, 20 were retained in a final model which explained 75.79% of the variance accounted for by a Rasch model. Among the components specified by the composite model, OP and READ were found to have the most influence, with RD exhibiting a moderate influence and SEARCH a low influence. Implications for developers of FCE include the need to consider and balance test method effects, and for other developers the additional need to determine whether their tests test features found to be criterial to the target level (such as non-standard word order at B2 level). Researchers wishing to use Khalifa and Weir’s (2009) model of reading should modify the stage termed named inferencing and consider adding further stages which define the way in which the goal setter and monitor work and the way in which item responses are selected. Finally, for those researchers interested in adopting a similar approach to that of the current study, careful consideration should be given to the way in which attributes are selected. The aims and scope of the study are of prime importance here.

O objetivo desta pesquisa foi aplicar ao Protocolo de Avaliação de Habilidades Linguístico- Pragmáticas de Gerber & Gurland (1989) em falantes de português brasileiro, descrevendo o comportamento linguístico exibido pelos participantes da amostra, e verificando se o desempenho dos voluntários pode ser influenciado por fatores de estilo, trajetória de vida e saúde, relatados em situação interacional a partir da aplicação de um questionário. Este protocolo avalia a ocorrência de quebras conversacionais de fundo linguístico ou pragmático e as estratégias de reparação dessas quebras, em interações dialógicas entre pacientes afásicos e interlocutores não afásicos, a partir da perspectiva do interlocutor, e possui poucos estudos na nossa população falante de português brasileiro. Partindo de uma perspectiva de língua enquanto função cognitiva do cérebro investigou-se se fatores de estilo, trajetória de vida e saúde por poderem afetar a cognição humana também afetariam o desempenho dos participantes. Foram realizadas e analisadas 66 entrevistas gravadas de participantes não afásicos com idade entre 18 e 75 anos de idade, distribuídos em três grupos etários, falantes de português brasileiro, com ensino fundamental completo como escolaridade mínima. As entrevistas foram guiadas por um Questionário de Informações Básicas elaborado pela autora a partir do Modelo STAC-R de Reuter-Lorenz & Park (2014) sobre fatores protetivos e depletivos à cognição humana. As análises de entrevistas que consideravam participantes sem vantagens e/ou desvantagens linguísticas/cognitivas mostraram que as quebras linguísticas apresentaram a seguinte ocorrência: quebras fonológicas mín.: 0, máx.: 3 vezes, quebras por problemas de recuperação lexical, mín.: 0, máx.: 1 vez e quebras por problemas semânticos, mín.: 0, máx.: 1 vez. Com relação às quebras pragmáticas, observou-se o seguinte padrão de ocorrência: por problemas de pressuposição/referência, mín.: 0, máx.: 4, problemas de mudança de turno, mín.: 0, máx.: 1, e no caso de quebras por manutenção de tópico não foram registradas ocorrências nas análises com a presença dos filtros de vantagem/desvantagem. O número de quebras detectadas sem discriminação do tipo de quebra foi mín.: 0, máx.: 5. A utilização de estratégias de reparação do indivíduo na análise em participantes sem vantagens/desvantagens apresentou a seguinte ocorrência: estratégia de reconhecimento, mín.: 0, máx.: 1, repetições:, mín.: 0, máx.: 3, paráfrase, mín.: 0, máx.: 1, adição de informação, mín.: 0, máx.: 3, revisão sintáticosemântica: mín.: 0, máx.: 2. As estratégias de reparação do indivíduo ocorreram no mínimo 0 e no máximo 7 vezes, em situações de ocorrência de quebra. Os resultados também mostraram que apenas a presença de problemas neurológicos foi estatisticamente significante para a ocorrência de quebras linguísticas e para o total de quebras detectadas na amostra. Esse resultado está dentro do esperado pois o protocolo foi elaborado para avaliação de afasia, que é um problema neurológico que afeta a linguagem especificamente. Entretanto, os problemas relatados pelos participantes não incluíam afasia, foram pontuais e bastante anteriores ao momento da entrevista na maior parte das vezes. Concluiu-se, portanto, que o protocolo é um bom instrumento de avaliação individual, sensível à presença de problemas neurológicos em nossa população também. Sugerimos que o protocolo seja utilizado de forma complementar a outros tipos de instrumentos de avaliação de linguagem, por ser específico para a avaliação da comunicação dialógica. / The aim of this research was to apply the Assessment Protocol of Pragmatic -Linguistic Skills of Gerber & Gurland (1989) in Brazilian Portuguese speakers, describing the linguistic behavior exhibited by the sample participants, and checking if the protocol performance could be influenced by lifestyle, life course and health factors reported in an interactional situation from the application of a questionnaire. This protocol evaluates the occurrence of linguistic or pragmatic conversational breakdowns and the strategies for repairing these breakdowns, in dialogic interactions between aphasic patients and non-aphasic interlocutors, from the perspective of the interlocutor, and has few studies in our Brazilian Portuguese speakers population. Based on a perspective of language as a cognitive function of the brain, we investigated whether factors related to lifestyle, life course and health because they could affect human cognition would also affect participants\' performance. Were carried out and analyzed, 66 recorded interviews with non - aphasic participants aged 18 to 75 years old, divided into three age groups, monolingual or bilingual speakers of Brazilian Portuguese, with complete primary education1 as minimum education. The interviews were guided by a Basic Information Questionnaire elaborated by the author based on the STAC-R Model of Reuter-Lorenz & Park (2014) on protective and depletive factors to human cognition. The interviews analysis that considered participants without linguistic/cognitive advantages and/or disadvantages showed that linguistic breakdowns presented the following occurrence: phonological breakdowns, min.: 0, max.:3 times, word retrieval breakdowns, min.: 0, max.: 1, and syntactic-semantic breakdowns, min.: 0, max.: 1. Within relation to pragmatic breakdowns, it was observed the following occurrence pattern: for pressupostional/referencial problems: min.: 0, max.: 4, turntaking problems, min.: 0, max.: 1 and in the case of topic maintenance breakdowns, it was not registered occurrences in the analysis. The number of breakdowns without discrimination of the type of the breakdown was min.: 0, max.: 5. The client repair strategies used in the analysis of participants without advantages/disadvantages presented the following occurrence: acknowledgment strategy, min.: 0, max.: 1, repetition, min.: 0, max.: 3, paraphrase, min.: 0, max.: 1, adding information, min.: 0, max.: 3, semantic-syntactic revision, min.: 0, max.: 2. The client repair strategies occurred in a minimum 0 and in a maximum 7 times in breakdown situations. The results also showed that only the presence of neurological problems was statistically significant for the occurrence of linguistc breakdowns and for the total number of breakdowns detected in the sample. This result is within the expected range, since the protocol was designed to evaluate aphasia, which is a neurological problem that affects language specifically. The problems reported by the participants did not include aphasia and were punctual and well before the moment of the interview most of the time. Therefore, it was concluded that the protocol is a good instrument for individual assessment, also sensitive to the presence of neurological problems in our population. We suggest that the protocol should be used in a complementary way to other types of language assessment tools, inasmuch it is specific for the evaluation of dialogic communication.