This thesis reports on the development of an Australian Government English as a Second Language test for health professionals, the Occupational English Test (OET) , and its validation using Rasch Item Response Theory models. / The test contains sub-tests of the four macroskills, each based on workplace communication tasks. The thesis reports on the creation of test specifications, the trial ling of test materials and the analysis of data from full test sessions. / The main research issues dealt with are as follows: / 1. The nature of the constructs involved in communicative language testing. The term proficiency is analysed, and its relationship to a number of models of communicative competence examined. The difficulty of incorporating into these models factors underlying test performance is identified. / 2. The nature of performance tests. A distinction is introduced between strong and weak senses of the term performance test, and related to the discussion in 1 above. / 3. The content validity of the OET. This is established on the basis of a questionnaire survey, interviews, examination of relevant literature, workplace observation and test data. / 4. The role of classical and Rasch IRT analysis in establishing the qualities of the test. Classical and Rasch IRT analyses are used to establish the basic reliability of the OET sub-tests. The Writing sub-test is shown to be somewhat problematic for raters because of the nature of the writing task involved. Analysis of data from the Reading subtest demonstrates the superiority of the Rasch analysis in the creation of short tests with a specific screening function. / 5. The role of Rasch IRT analysis in investigating the construct and content validity of the test and hence of communicatively-oriented tests in general. Rasch analysis reveals that the sub-tests are satisfactory operationalizations of the constructs 'ESL listening/ speaking/ reading/ writing ability in health professional contexts. For the Speaking and Writing sub-tests, the analysis reveals that responses of raters in categories associated with perceptions of grammatical accuracy have a more important role in the determination of the candidate's total score than was anticipated in the design of the test. This finding has implications for the validity of communicatively oriented tests in general, and illustrates the potential of IRT analysis for the investigation of the construct validity of tests. / 6. The appropriateness of the use of Rasch IRT in the analysis of language tests. The nature of the debate about 'unidimensionality' in Rasch analysis is reviewed. It is argued that the issue has been substantialy misunderstood. Data from the two parts of the Listening sub-test are analysed, and statistical tests are used to confirm the unidimensionality of the data set. It is concluded that Rasch analysis is appropriate for a language test of this type. / 7. The behaviour of raters in the rating of oral and written production in a second language. The findings reported in 5 above suggest that the behaviour of raters is crucial to understanding what is being measured in a communicative test of the productive language skills. / The research demonstrates the value of Rasch IRT analysis in the empirical validation of communicatively oriented language tests, and the potential of large-scale test development projects for theoretical work on language testing.
| Identifer | oai:union.ndltd.org:ADTP/275865 |
| Date | January 1990 |
| Creators | McNamara, Timothy Francis |
| Source Sets | Australiasian Digital Theses Program |
| Language | English |
| Detected Language | English |
| Rights | Terms and Conditions: Copyright in works deposited in the University of Melbourne Eprints Repository (UMER) is retained by the copyright owner. The work may not be altered without permission from the copyright owner. Readers may only, download, print, and save electronic copies of whole works for their own personal non-commercial use. Any use that exceeds these limits requires permission from the copyright owner. Attribution is essential when quoting or paraphrasing from these works., Open Access |
Page generated in 0.0019 seconds