This three-article dissertation addresses three different yet interrelated topics: language testing, assessment, and evaluation. The first article (Saudi Student Placement into ESL Program Levels: Issues beyond Test Criteria) addresses a crucial yet understudied issue concerning why lower-level ESL classes typically contain a disproportionate number of Saudi students. Based on data obtained from different stakeholders, the findings revealed that one-third of the study students intentionally underperformed on ESL placement tests. However, ESL administrators participating in this study provided contradicting findings. The second article explores the efficacy of (Integrating Self-assessment Techniques into L2 Classroom Assessment Procedures) by examining the accuracy of CEFR self-assessment rubric compared to students' TOEFL scores, and the extent to which gender and levels of language proficiency cause any potential score underestimation. By obtaining data from 21 ESL students attending the Center for English as a Second Language at University of Arizona, the findings revealed no statistically significant correlations between participants' self-assessed scores and their TOEFL scores. However, the participants reported that the CEFR self-assessment rubric is accurate in measuring their levels of language proficiency. On the other hand, the third article (Quality Assurance and Accreditation as Forms for Language Program Evaluation: A Case Study of Two EFL Departments in A Saudi University) provides a simulated program evaluation based on an integrated set of standards of the NCAAA (the National Commission for Academic Accreditation and Assessment) and CEA (the Commission on English Language Program Accreditation). The findings indicated that the standards of the mission, curriculum, student learning outcomes, and program development, planning, and review, were partially met, whereas the standards of teaching strategies, assessment methods, and student achievement were not.

Diss. Göteborg : Göteborgs universitet, 2009.

Thesis (Ph. D.)--University of Texas at Austin, 2007. / Vita. Includes bibliographical references.

This study explores, from a test validity perspective, the extent to which the two variables of topic and background knowledge of topic have an effect on spoken performance in language proficiency interviews. It is argued that in assessment contexts where topics are randomly assigned to test takers, it is necessary to demonstrate that topics of tasks and the level of background knowledge that test takers brings to these topics do not exert an undue influence on test results. Otherwise, a validity threat may be introduced to the test. Data were collected from 82 Farsi speakers of English who performed on ten different topics, across three task types. Participants’ background knowledge of topics was elicited using self- report questionnaires while C-tests were used as a measure of general English language proficiency. Four raters assigned scores to spoken performances using rating scales. Semi- structured interviews were carried out with raters, upon completion of the rating process. A mixed- methods strategy of inquiry was adopted where findings from the quantitative analyses of score data (using Multi-Faceted Rasch Measurement, multiple regression and descriptive statistics) were synthesised with the results of the qualitative analyses of rater interviews and test takers’ content of speech in addressing the foci of the study. The study’s main findings showed that the topics used in the study exhibited difficulty measures which were statistically distinct i.e. topics, within a given task type, could not be considered parallel. However, the size of the differences in topic difficulties was too small to have a large practical effect on scores. Participants’ different levels of background knowledge were shown to have a consistent, systematic and statistically significant effect on performance with low levels of background knowledge posing the highest level of challenge for test takers and vice versa. Nevertheless, these statistically significant differences in background knowledge levels failed to translate into practically significant differences, as the size of the differences were too small to have a large impact on performance scores. Results indicated that, compared to general language proficiency which accounted for approximately 60% of the variance in spoken performance scores, background knowledge only explained about 1-3% of the variance. Qualitative analyses of data suggested lack of background knowledge to be associated with topic abandonment, disengagement from topic-related questions, and fewer opportunities for test takers to elaborate on topics. It was also associated with negative affective influence on test takers, particularly lower proficiency individuals. Taken together, the findings have theoretical, methodological and practical implications for second language speaking performance assessment.

This present study investigates the development of second language (L2) Chinese learners’ writing by 1) subjective ratings of essay quality, 2) a battery of objective measures representing the general syntactic complexity as well as specific syntactic features, and 3) the sources of verb phrase complexity used by learners of different institutional levels. This study first compares the subjective ratings of the essays written by learners across four institutional levels and then uses Cumulative Linked Model to examine the contribution of the objective measures of linguistic features to the essay ratings. This study further identifies a number of sources used by learners to construct complex verb phrases, which is an important contributor of the essay rating, and compares the amount of usages by learners at different institutional levels. The purpose of the study is to better understand L2 Chinese learners’ syntactic development in writing from multi-dimensional perspectives, and to identify the most crucial elements that determine the quality of writing. This study recruits 105 L2 Chinese college learners to write a narrative essay and an argumentative essay according to the prompts. Each of the writing sample is rated by two independent raters according to the holistic ACTFL Proficiency Guidelines, as well as the analytic rubric which was adapted from the ESL Composition Profile for this study. The derivation of syntactic complexity measures was based on the rank scales of lexicogrammar in Systemic Functional Linguistics (Halliday & Matthiessen, 2014), involving 12 features at the levels of clause complex, clause, and verb phrase, some of which represent constructions unique to Chinese. A series of statistical tests, including Kruskal-Wallis tests, Dunn’ tests, Spearman’ correlation tests, and CLM are performed to answer that research questions. The findings show that 1) learners’ overall writing quality measured by holistic and analytic ratings do not show significant differences across the first several academic years; 2) higher-level learners are more heterogeneous in writing ability than lower-level learners; 3) phrasal complexity contributes more to the essay quality than clausal complexity; 4) syntactic complexity features that learners develop fastest hardly overlap with those that contribute most to the essay rating; 5) complex verbal phrases come from 10 different sources and the composition of complex verbal phrases remain stable across the groups; and 6) essay types makes significant differences in terms of holistic and analytic ratings, use of syntactic complexity features, as well as their contribution to the essay ratings. From the pedagogical view, this study points out that instruction should focus more on complexity at the phrasal level, especially nominalization and complex verb phrases, that play a more important role to determine the writing quality. Some of the current focus in instruction may not necessarily lead to better quality or higher proficiency in Chinese writing.

This dissertation investigates how L2 speakers’ assessment of their own pronunciation compares to the assessment of these speakers’ pronunciation by different types of listeners. Study 1 investigated the associations between L2 speakers’ pronunciation self-assessment and the assessment by L1 listeners. Eighty-two L2 English speakers performed a picture narrative task and rated their own speech. These speech samples were also rated by eight inexperienced L1 English listeners. Pearson correlation and paired t-test analyses revealed that the speakers’ self-assessment was significantly different from L1 English listeners’ assessment, and that poor performers overestimated their performance while top performers underestimated it. Study 2 investigated the associations between L2 speakers’ pronunciation self-assessment and the assessment by L1 listeners, L2 listeners who shared an L1 with the speakers, and L2 listeners who did not share an L1 with the speakers. Forty-one L1 Mandarin speakers performed a picture narrative task in English and rated their own pronunciation. These speech samples were also rated by L1 English listeners, L1 Mandarin listeners, and L1 mixed listeners. Pearson correlation and paired t-test analyses revealed that the alignment between self- and other-assessment varied according to the L1 background of the listeners and the construct under evaluation. Study 3 investigated if L2 listeners had an advantage over L1 listeners at comprehending L2 speech, and if the L1 background and proficiency level of the L2 speakers and listeners affected this potential advantage. Forty-one Mandarin-accented English speech samples from a picture narrative task were rated for comprehensibility by three groups of listeners – L1 English listeners, L1 Mandarin listeners, and L1 mixed listeners. Paired t-test analyses revealed that L1 Mandarin listeners perceived the Mandarin-accented speech to be more comprehensible than the L1 English listeners did, and this benefit was observed with three different proficiency combinations when proficiency was taken into consideration. Although overall the L1 mixed listeners did not perceive the Mandarin-accented speech to be more comprehensible than the L1 English listeners did, when proficiency was taken into consideration, the picture was more complex – while a comprehensibility benefit was observed with one specific proficiency combination, a comprehensibility detriment was observed with a different proficiency pairing.

Researchers have intermittently treated the topic of the language in which reading comprehension test questions should be presented in, or language of assessment (LoA). The overall consensus has been that questions in L1 lead to better scores and that questions in L1 should be used for reading comprehension particularly at the beginning levels. However, minimal research has been conducted at the advanced level, and no research has been found where proficiency items, empirically validated, were utilized in testing instruments. Furthermore, explanatory data from qualitative analysis has been sparse. The present research endeavored to satisfy these areas of needed research. It was found that a group of advanced learners of Russian performed better when MC questions were presented in English. Student attitudes, as revealed by survey items, depicted questions in L2 as more difficult. It was additionally found that vocabulary was one of the major factors in difficulty. Matters pertaining to validity and face validity surfaced and were designated, along with gathering other qualitative data, as the recommended direction of future research.

Submitted by Fernanda Ciolfi (fernanda.ciolfi@puc-campinas.edu.br) on 2016-08-11T14:56:16Z No. of bitstreams: 1 RAUNI JANDRE ROAMA ALVES.pdf: 4199544 bytes, checksum: 2f1f3a6d003fd985c7aa1410d2915ea8 (MD5) / Made available in DSpace on 2016-08-11T14:56:16Z (GMT). No. of bitstreams: 1 RAUNI JANDRE ROAMA ALVES.pdf: 4199544 bytes, checksum: 2f1f3a6d003fd985c7aa1410d2915ea8 (MD5) Previous issue date: 2016-05-16 / Conselho Nacional de Pesquisa e Desenvolvimento Cient?fico e Tecnol?gico - CNPq / Nationally there is a dearth of adequate psychometric instruments that assess risk of Developmental Dyslexia (DD), which is why the "Identifying Signs of Dyslexia Test" (TISD) was prepared. This study aimed to investigate evidence of validity based on external variables and internal structure, as well as reliability of TISD. Participated children and adolescents of both genders, aged 6 to 16, divided into two samples: (1) Group Case: 371 from the group ?Neuro- Learning Difficulties? of the Clinical Hospital of the State University of Campinas, and 12 diagnosed, at the end of the evaluation process, with DD; (2) Group No Case: 288 children without complaints of learning difficulties, with 112 coming from private schools and 151 from public school both located in the state of S?o Paulo, and 25 students of a public school from a city in Rio Grande do Norte. The following instruments were applied: (1) TISD: evaluates academic and neuropsychological skills related to reading, consists of 8 subtests: Reading, Writing, Visual Attention, Motor Skills, Calculation, Phonological Awareness, Rapid Naming and Short-term Memory; (2) Academic Performance Test (TDE): evaluates school capacity and is composed of subtests of Reading, Writing and Arithmetic; (3) Test Bender-Gestalt (B-SPG): evaluates perceptomotora maturity, through the analysis of the distortion of fashion model figures; (4) Wechsler Intelligence Scale for Children, 3rd and 4th Edition (WISC-III / WISC-IV): evaluates intelligence by benchmarking cognitive abilities, that only the Digits, Cancellation and Arithmetic subtests were used. To achieve the goals, six studies were conducted. The studies of evidences of validity based on external variables relations indicated the following results: (1) age and type of school influences in total performance test; the instrument was sensitive to differentiate the performance of individuals aged six, seven and eight, with worst performing of public school; (2) the search for evidence of convergent validity, the TISD was correlated with other validated instruments for the Brazilian population (TDE; WISC III / IV; B-SPG); the results showed statistically significant and moderate correlations between the TISD subtests, Reading, Writing, Calculation, Working Memory, Motor Skills, with the TDE subtests of Reading and Writing, with the subtests Arithmetic and Digits of the WISC-IV and B-SPG, respectively, indicating convergent validity; (3) the TISD was able to differentiate the diagnosis group of DD and the group without learning difficulties complaints. The search study of evidence of validity based on the internal structure indicated a two-factor model for TISD (without the inclusion of subtest Rapid Naming) with a factor composed of neuropsychological skills (subtests of Motor Skills, Visual Attention, Phonological Awareness and Short-term Memory) and another for school skills (Reading, Writing and Calculation subtests). Precision studies indicated the following results: all items / tasks in each subtest showed significant correlations with the total result of the own subtest, ranging from moderate to strong correlations. It was found that the studies reported here indicate favorable results for the continuity of psychometric studies of TISD and contributed to the advancement of the stages of its development. / Nacionalmente verifica-se grande escassez de instrumentos psicom?tricos adequados para a avalia??o para risco de Dislexia do Desenvolvimento (DD), motivo pelo qual o ?Teste de Identifica??o de Sinais de Dislexia? (TISD) foi elaborado. Este trabalho teve como objetivo investigar evid?ncias de validade baseadas em vari?veis externas e na estrutura interna e precis?o do TISD. Participaram crian?as e adolescente de ambos os g?neros, com idade entre 6 e 16 anos, divididos em duas amostras: (1) Grupo Caso: 371 pertencentes ao Ambulat?rio de Neuro-Dificuldades de Aprendizagem do Hospital de Cl?nicas da Universidade Estadual de Campinas, sendo 12 com DD; (2) Grupo N?o Caso: 288 crian?as sem queixas de dificuldades de aprendizagem, sendo 112 de escola particular e 151 de escola p?blica do interior do estado de S?o Paulo, e 25 de uma escola p?blica de uma cidade do Rio Grande do Norte. Foram aplicados os seguintes instrumentos: (1) TISD: avalia habilidades acad?micas e neuropsicol?gicas relacionadas ? leitura, composto por 8 subtestes: Leitura, Escrita, Aten??o Visual, Habilidades Motoras, C?lculo, Consci?ncia Fonol?gica, Nomea??o R?pida e Mem?ria Imediata; (2) Teste de Desempenho Escolar (TDE): avalia capacidades escolares, ? composto pelos subtestes de Leitura, Escrita e Aritm?tica; (3) Teste Gest?ltico Visomotor de Bender (B-SPG): avalia maturidade perceptomotora, por meio da an?lise da distor??o da forma de figuras modelo; (4) Escalas Wechsler de Intelig?ncia para Crian?as, 3? e 4? edi??o (WISC-III/WISC-IV): avalia intelig?ncia pelo aferimento de habilidades cognitivas, das quais foram utilizados os subtestes D?gitos, Cancelamento e Aritm?tica. Os estudos de busca de evid?ncias de validade baseadas em rela??es com vari?veis externas indicaram os seguintes resultados: (1) influ?ncias de idade e tipo escolas no desempenho total do teste; o instrumento se mostrou sens?vel na identica??o de idades de seis, sete e oito anos, com pior desempenho da escola p?blica; (2) o TISD foi correlacionado a outros instrumentos validados para a popula??o brasileira (TDE; WISC III/IV; B-SPG); os resultados mostraram correla??es estatisticamente significativas e moderadas entre os subtestes Leitura, Escrita, C?lculo, Mem?ria de Trabalho, Habilidades Motoras do TISD com os subtestes Leitura e Escrita do TDE, subtestes Aritm?tica e D?gitos da WISC-IV e B-SPG, respectivamente, indicando validade convergente; (3) o TISD foi capaz de diferenciar o grupo diagn?stico da DD de crian?as sem queixas de dificuldades de aprendizagem. O estudo de busca de evid?ncias de validade baseadas na estrutura interna indicou um modelo bi-fatorial para o TISD (sem a inclus?o do subteste de Nomea??o R?pida), com um fator composto por habilidades neuropsicol?gicas (subtestes de Habilidades Motoras, Aten??o Visual, Consci?ncia Fonol?gica e Mem?ria de Curto Prazo) e outro por habilidades escolares (subtestes de Leitura, Escrita e C?lculo). Os estudos de precis?o indicaram os seguintes resultados: todos os itens/tarefas de cada subteste apresentaram correla??es significativas com o resultado total do teste, oscilando entre correla??es moderadas e fortes. Verificou-se que a maioria dos estudos aqui realizados indicou resultados favor?veis para a continuidade dos estudos psicom?tricos do TISD e contribu?ram para o avan?o das etapas de sua elabora??o.

The process of migration has resulted in population growth and contributed to the transformation of New Zealand. Migrant and refugee children face many adjustment factors and their ease in resettling in New Zealand is largely dependent on their ability to learn English. Migration stress, change, trauma and loss may result in psychological difficulties which in turn may affect their resettling and learning. The Ministry of Education and other professionals work together to enhance the quality of their service provision to facilitate easier adjustment, resettlement and effective learning for these children. An adequate assessment battery for speech language therapists to assess migrant and refugee children, is presently lacking in New Zealand. Therapists currently use various assessments, with the assistance of interpreters. The New Zealand Speech Therapists’ Association (NZSTA), in accordance with speech therapists in Group Special Education (GSE), strongly supports the need for research with these groups and the development of an appropriate assessment battery. This exploratory study aimed to determine an assessment battery for use in examining English language acquisition in refugee and migrant children and to highlight the benefit of using measurement tools that determine incremental change over time in contrast to the use of monolingual psychometric tests. The study explored a selected assessment battery and gathered data in five main focus areas, namely: cognition, language, trauma, classroom behaviour, developmental and birth information. Eligible children were those who did not have physiologically - impaired cognitive abilities. Eight cases, four refugee and four migrant students, were selected by convenience sampling. All participants were children selected from primary school 1 (three refugees and three migrants) and primary school 2 (one refugee and one migrant) primary schools, aged approximately (5-8 years). Participants included four male and four females, refugee and migrant children, and those with both high and low English ability. Based on the study’s results, recommendations were made to refine the test battery, which included test modification. For example, the use of the trauma measurement tool only if there is prior evidence of trauma, the inclusion of a larger test population who have a common primary language to allow for cost effective interpreter use and to also allow for generalisations to be made, the inclusion of an assessment of the children’s primary language in order to determine the relationship, development and acquisition of the child’s second language with reference to his/her development and skills in his native language. All of the refugee children and 3 migrant children displayed slower processing time during the administration of the tests. Migrant parents were quicker in test completion as compared to refugee parents. They displayed differences in family size, contact with extended family, socioeconomic status and educational level. Migrant children produced sentences that included correct word order and sequence whilst refugee children produced sentences that lacked adequate word order or lacked articles and determiners. The study found the proposed test battery was an effective choice for use in the assessment of both migrant and refugee children, as the battery allows for dynamic assessment of children from diverse groups and this proved to be an unbiased means of assessing their English language and cognitive skills. Recommendations are made for future, more-extensive research. These findings provide information about appropriate and reliable language acquisition tests that measure incremental change with time. This study will contribute to a developing knowledge base for speech-language therapists who work with migrant or refugee children. Effective assessment on which to base tailored language programmes will assist them to optimise their experience in New Zealand schools and enhance their English language skills.

The process of migration has resulted in population growth and contributed to the transformation of New Zealand. Migrant and refugee children face many adjustment factors and their ease in resettling in New Zealand is largely dependent on their ability to learn English. Migration stress, change, trauma and loss may result in psychological difficulties which in turn may affect their resettling and learning. The Ministry of Education and other professionals work together to enhance the quality of their service provision to facilitate easier adjustment, resettlement and effective learning for these children. An adequate assessment battery for speech language therapists to assess migrant and refugee children, is presently lacking in New Zealand. Therapists currently use various assessments, with the assistance of interpreters. The New Zealand Speech Therapists’ Association (NZSTA), in accordance with speech therapists in Group Special Education (GSE), strongly supports the need for research with these groups and the development of an appropriate assessment battery. This exploratory study aimed to determine an assessment battery for use in examining English language acquisition in refugee and migrant children and to highlight the benefit of using measurement tools that determine incremental change over time in contrast to the use of monolingual psychometric tests. The study explored a selected assessment battery and gathered data in five main focus areas, namely: cognition, language, trauma, classroom behaviour, developmental and birth information. Eligible children were those who did not have physiologically - impaired cognitive abilities. Eight cases, four refugee and four migrant students, were selected by convenience sampling. All participants were children selected from primary school 1 (three refugees and three migrants) and primary school 2 (one refugee and one migrant) primary schools, aged approximately (5-8 years). Participants included four male and four females, refugee and migrant children, and those with both high and low English ability. Based on the study’s results, recommendations were made to refine the test battery, which included test modification. For example, the use of the trauma measurement tool only if there is prior evidence of trauma, the inclusion of a larger test population who have a common primary language to allow for cost effective interpreter use and to also allow for generalisations to be made, the inclusion of an assessment of the children’s primary language in order to determine the relationship, development and acquisition of the child’s second language with reference to his/her development and skills in his native language. All of the refugee children and 3 migrant children displayed slower processing time during the administration of the tests. Migrant parents were quicker in test completion as compared to refugee parents. They displayed differences in family size, contact with extended family, socioeconomic status and educational level. Migrant children produced sentences that included correct word order and sequence whilst refugee children produced sentences that lacked adequate word order or lacked articles and determiners. The study found the proposed test battery was an effective choice for use in the assessment of both migrant and refugee children, as the battery allows for dynamic assessment of children from diverse groups and this proved to be an unbiased means of assessing their English language and cognitive skills. Recommendations are made for future, more-extensive research. These findings provide information about appropriate and reliable language acquisition tests that measure incremental change with time. This study will contribute to a developing knowledge base for speech-language therapists who work with migrant or refugee children. Effective assessment on which to base tailored language programmes will assist them to optimise their experience in New Zealand schools and enhance their English language skills.