Despite Landy and Farr’s (1980) highly influential call for a moratorium on rating format research over 30 years ago, recent research continues to show that rating format design can influence rating quality. This symposium brings together a group of scholars who will share their research and expertise on this topic.

No description available.

<p>The diversity embodying South Africa has emphasized the importance and influence of language in education and thus the additive bilingual programme is being implemented in the Eastern Cape by the ABLE project in order to realize the South African Language in education policy (LEiP). In accordance with this, the Woodcock Munoz Language Survey (which specializes in measuring cognitive academic language proficiency) was chosen as one of the instruments to evaluate the language outcomes of the programme and was adapted into South African English and isiXhosa. The current study was a subset of the ABLE project, and was located within the bigger project dealing with the translation of the WMLS into isiXhosa and the successive research on the equivalence of the two language versions. This study evaluated the structural equivalence of the English and isiXhosa versions of the WMLS on matched sample groups (n= 150 in each language group). Thus secondary data analysis (SDA) was conducted by analyzing the data in SPSS as well as CEFA (Comprehensive Exploratory Factor Analysis). The original data set was purposively sampled according to set selection criteria and consists of English and isiXhosa first language learners. The study sought to confirm previous research by cross-validating the results of structural equivalence on two subscales, namely the Verbal Analogies (VA) and Letter-Word Identification (LWI) subscale. The research design reflects psychometric test theory and is therefore located in a bias and equivalence theoretical framework. The results of the exploratory factor analysis found that one can only accept structural equivalence in the first factor identified in the VA subscale, while structural equivalence was found in the factor for the LWI subscale. The use of scatter-plots to validate the results of the exploratory factor analysis indicated that one can tentatively accept these results. The study thus contributed to the literature on the translation of the WMLS, and the adaptation of language tests into the indigenous languages of South Africa,as well as additive bilingual programmes.</p>

<p>The diversity embodying South Africa has emphasized the importance and influence of language in education and thus the additive bilingual programme is being implemented in the Eastern Cape by the ABLE project in order to realize the South African Language in education policy (LEiP). In accordance with this, the Woodcock Munoz Language Survey (which specializes in measuring cognitive academic language proficiency) was chosen as one of the instruments to evaluate the language outcomes of the programme and was adapted into South African English and isiXhosa. The current study was a subset of the ABLE project, and was located within the bigger project dealing with the translation of the WMLS into isiXhosa and the successive research on the equivalence of the two language versions. This study evaluated the structural equivalence of the English and isiXhosa versions of the WMLS on matched sample groups (n= 150 in each language group). Thus secondary data analysis (SDA) was conducted by analyzing the data in SPSS as well as CEFA (Comprehensive Exploratory Factor Analysis). The original data set was purposively sampled according to set selection criteria and consists of English and isiXhosa first language learners. The study sought to confirm previous research by cross-validating the results of structural equivalence on two subscales, namely the Verbal Analogies (VA) and Letter-Word Identification (LWI) subscale. The research design reflects psychometric test theory and is therefore located in a bias and equivalence theoretical framework. The results of the exploratory factor analysis found that one can only accept structural equivalence in the first factor identified in the VA subscale, while structural equivalence was found in the factor for the LWI subscale. The use of scatter-plots to validate the results of the exploratory factor analysis indicated that one can tentatively accept these results. The study thus contributed to the literature on the translation of the WMLS, and the adaptation of language tests into the indigenous languages of South Africa,as well as additive bilingual programmes.</p>

The diversity embodying South Africa has emphasized the importance and influence of language in education and thus the additive bilingual programme is being implemented in the Eastern Cape by the ABLE project in order to realize the South African Language in education policy (LEiP).In accordance with this, the Woodcock Munoz Language Survey (which specializes in measuring cognitive academic language proficiency) was chosen as one of the instruments to evaluate the language outcomes of the programme and was adapted into South African English and isiXhosa.The current study was a subset of the ABLE project, and was located within the bigger project dealing with the translation of the WMLS into isiXhosa and the successive research on the equivalence of the two language versions. This study evaluated the structural equivalence of the English and isiXhosa versions of the WMLS on matched sample groups (n= 150 in each language group). Thus secondary data analysis (SDA) was conducted by analyzing the data in SPSS as well as CEFA (Comprehensive Exploratory Factor Analysis). The original data set was purposively sampled according to set selection criteria and consists of English and isiXhosa first language learners. The study sought to confirm previous research by cross-validating the results of structural equivalence on two subscales, namely the Verbal Analogies (VA) and Letter-Word Identification (LWI) subscale. The research design reflects psychometric test theory and is therefore located in a bias and equivalence theoretical framework. The results of the exploratory factor analysis found that one can only accept structural equivalence in the first factor identified in the VA subscale, while structural equivalence was found in the factor for the LWI subscale.The use of scatter-plots to validate the results of the exploratory factor analysis indicated that one can tentatively accept these results. The study thus contributed to the literature on the translation of the WMLS, and the adaptation of language tests into the indigenous languages of South Africa,as well as additive bilingual programmes. / Magister Artium (Psychology) - MA(Psych)

This dissertation concerns oral language tests from a grammatical perspective. Tests today are usually assessed based on a communicative approach to language, so it is interesting to see how great a significance the level of grammatical development of test takers has for their communicative competence in general. The data in the investigation consist of recorded test conversations from a Swedish national language proficiency test, Tisus (test in Swedish for university and university college studies). The general aim of the study is to investigate whether there is a difference between the test takers who passed and those who failed in terms of their level of grammatical development. This is investigated in one main study and two smaller follow-up studies. The theoretical basis for this work is comprised of theories on second language learning and theories on language testing. For the grammatical analysis, Pienemann’s processability theory (PT) is applied. This theory posits that learners acquire certain morpho-syntactic structures in a particular order, something that produces a hierarchy consisting of five levels. In the main study, a quantitative analysis is made of the test takers’ level of grammatical development relative to their TISUS results. The results show a clear correlation between grammatical level and test results. There also seems to be a kind of watershed at level 4, given that most of the test takers who passed have command of this level in the PT hierarchy, whereas those who failed, as a rule, only make it to level 3. The first follow-up study is more qualitative in nature and is focused on the test takers who deviate from the general pattern of results in the main study: those who failed despite their high grammatical level and those who passed despite their low grammatical level. One result of this follow-up study is that, in these cases, it appears communicative competence was critical. Being able to orient oneself to the special conversational situation represented by the test conversation is considered to be particularly important. In a second follow-up study, the problems of applying PT to morphology and syntax are examined. The results of the dissertation suggest that there is a correlation between grammatical competence and communicative competence in general. This means that the grammatical levels in the PT hierarchy may constitute a useful basis of analysis in oral assessments, as a complement to other assessment tools.

Doctorat en sciences psychologiques / info:eu-repo/semantics/nonPublished

The study reported in this thesis investigated Emirati students’ claims related to experiences of cultural bias of the reading component of the IELTS examination through a critical realist lens. Critical realism posits a layered reality which allows for the conceptualization of experiences as emerging from the interplay of events and mechanisms found in two other realms of reality – the actual and the real. Experiences, therefore, have a different ontological status than the events and the causal mechanisms to which they are attributed. Social realism was used to further explore the depth of the realm of the real through Archer’s construct of analytical dualism. This allowed for the placement of generative mechanisms into three domains: structure, culture, and agency. There were two parts to this investigation: a content analysis and a focus group study. The first part of the content analysis consisted of analyzing 60 reading passages from 20 IELTS examinations for a number of cultural categories. These included such things as cultural objects, social roles, idiomatic expressions, traditions and festivals, superstitions and beliefs, and political and historical settings. The second part of the content analysis focused on the question types and syntactical structure of the 5 different IELTS examinations that the focus groups students sat. All three components of the analysis – the cultural content, question types, and syntactical structure – were conducted at the level of the actual. Findings indicated that on average, an IELTS examination contained 14 cultural references of various kinds. Only 4% of all geographical references pertained to the Middle East with the biggest share being western locations. The most common question types were matching questions, fill in the blank questions, and yes/no questions with more than 50% of all questions requiring some form of higher order thinking due to text reinterpretation. The study also found that the question types were not consistently distributed over the examinations with each consisting of a different variety of questions and some even having repetitive question types on one reading examination. The second part of the study was the focus groups. Here, 21 Emirati students sat 5 different IELTS examinations. Upon test completion, these students underwent a semistructured interview to relate their experiences of the test. These experiences, at the level of the empirical, all shared 7 ideas: reading is hard, the questions are too difficult, the passages are too long and difficult, the topics are unfamiliar, the topics are not interesting, the vocabulary is too difficult, and there is not enough time. When the processes of retroduction and abduction were applied to both the content analysis and these common experiences, numerous structures and discourses at the level of the real were identified as having contributed to the emergence of the feeling of bias at the level of the empirical. These structures included such things as the students’ school system (eg. curriculum, assessment, instructors etc.), religion, literacy practices, and home. In the cultural domain, a number of discourses were found to contribute to the experiences at the level of the empirical. Amongst these were the ‘Unimportance of Reading’, the culture of ‘Obedience’, the rejection of the ‘un-Islamic’, and the students’ sense of ‘Entitlement’.

This study aims at exploring the quality of the Swedish and Norwegian national tests using their respective model texts for assessing. The study does so by relating them to the CEFR and the grading tool Write &amp; Improve within the context of the two countries and the field of language testing. The study finds there to be a set of inconsistences between what the national tests want to do and what they actually do. In particular, the study finds the Swedish national test not to be up to its own standards.

Orientador: Douglas Altamiro Consolo / Resumo: Neste trabalho investiga-se a abordagem de avaliação da proficiência oral do professor de línguas estrangeiras empregado por um exame brasileiro voltado a essa finalidade específica, o EPPLE – Exame de Proficiência para Professores de Línguas Estrangeiras. O estudo tem por objetivo propor uma abordagem que atenda a necessidades levantadas pela literatura da área acerca da importância da democratização e transparência do processo avaliativo e do emprego de processos decisórios que garantam os princípios de validade sem causar interferências na praticidade das avaliações. Assim, a abordagem proposta subordina-se à elaboração de uma escala de mensuração e à reestruturação da escala de proficiência do exame, agregando-se a opinião de usuários com diferentes graus de especialidade sobre a avaliação de proficiência. Trata-se, portanto, de um estudo de caso de natureza híbrida e cunho exploratório-descritivo que se utiliza de dados secundários e primários para auxiliar a investigação. Os dados secundários consistem em estudos e documentos sobre a caracterização da proficiência linguística para finalidades gerais e para finalidades de ensino, bem como sobre a avaliação dessas proficiências; em dados de fala do teste oral do EPPLE referentes à participação de formandos em Letras de uma universidade pública paulista e de duas universidades públicas mineiras, na qualidade de candidatos ao exame, nos anos de 2015 e 2017; e nos respectivos conceitos atribuídos aos candidatos, seguindo-se ... (Resumo completo, clicar acesso eletrônico abaixo) / Abstract: This research study investigates the assessment model used by a teaching-oriented language examination, the EPPLE – Proficiency Examination for Foreign Language Teachers, in the assessment of foreign language teachers’ oral proficiency. We aim at evaluating the model used by the oral test and proposing a new one that meets current concerns raised by the literature regarding the importance of democratizing the assessment process, guaranteeing transparency and using decision models that assure validity principles without causing interferences in the test’s practicality. Therefore, the model we propose relies on the reorganization of the oral proficiency scale used in the exam through the inclusion of its stakeholders’ opinions. The investigation conducted consists on a quali-quantitative and exploratory-descriptive case study that uses primary and secondary data. Secondary data consist of research studies and documents on the concept of general and teaching purpose language proficiency; speaking data from candidates to the oral test of the EPPLE examination; as well as the respective rates assigned to such candidates according to the proficiency scale used in the exam. The candidates to the EPPLE examination that participate in this study are Letters course undergraduate students from Brazilian public universities, one located in the state of São Paulo and two in the state of Minas Gerais, in the years of 2015 and 2017. Primary data, on the other hand, consist of the answers ob... (Complete abstract click electronic access below) / Doutor