This study explores, from a test validity perspective, the extent to which the two variables of topic and background knowledge of topic have an effect on spoken performance in language proficiency interviews. It is argued that in assessment contexts where topics are randomly assigned to test takers, it is necessary to demonstrate that topics of tasks and the level of background knowledge that test takers brings to these topics do not exert an undue influence on test results. Otherwise, a validity threat may be introduced to the test. Data were collected from 82 Farsi speakers of English who performed on ten different topics, across three task types. Participants’ background knowledge of topics was elicited using self- report questionnaires while C-tests were used as a measure of general English language proficiency. Four raters assigned scores to spoken performances using rating scales. Semi- structured interviews were carried out with raters, upon completion of the rating process. A mixed- methods strategy of inquiry was adopted where findings from the quantitative analyses of score data (using Multi-Faceted Rasch Measurement, multiple regression and descriptive statistics) were synthesised with the results of the qualitative analyses of rater interviews and test takers’ content of speech in addressing the foci of the study. The study’s main findings showed that the topics used in the study exhibited difficulty measures which were statistically distinct i.e. topics, within a given task type, could not be considered parallel. However, the size of the differences in topic difficulties was too small to have a large practical effect on scores. Participants’ different levels of background knowledge were shown to have a consistent, systematic and statistically significant effect on performance with low levels of background knowledge posing the highest level of challenge for test takers and vice versa. Nevertheless, these statistically significant differences in background knowledge levels failed to translate into practically significant differences, as the size of the differences were too small to have a large impact on performance scores. Results indicated that, compared to general language proficiency which accounted for approximately 60% of the variance in spoken performance scores, background knowledge only explained about 1-3% of the variance. Qualitative analyses of data suggested lack of background knowledge to be associated with topic abandonment, disengagement from topic-related questions, and fewer opportunities for test takers to elaborate on topics. It was also associated with negative affective influence on test takers, particularly lower proficiency individuals. Taken together, the findings have theoretical, methodological and practical implications for second language speaking performance assessment.
| Identifer | oai:union.ndltd.org:bl.uk/oai:ethos.bl.uk:600722 |
| Date | January 2013 |
| Creators | Khabbazbashi, Nahal |
| Contributors | Vanderplank, Robert ; Walter, Catherine |
| Publisher | University of Oxford |
| Source Sets | Ethos UK |
| Detected Language | English |
| Type | Electronic Thesis or Dissertation |
| Source | http://ora.ox.ac.uk/objects/uuid:359c8956-4561-43a8-a7ae-eba1e0dab51c |
Page generated in 0.002 seconds