This descriptive case study examined how highly experienced raters do writing assessment, with a focus on how raters defined the task under two conditions: (1) as external raters and (2) as 'teacher as rater'. Three raters followed a think-aloud procedure as they evaluated student writing. The semantic structure of the think-aloud protocols was analyzed via the Task Independent Coding method. This analysis yielded a detailed representation of the objects and operations used by raters. The sequence which raters followed as they used these objects and operations was represented schematically by problem behavior graphs for each scoring decision made (N = 360). Analyses of the problem behavior graphs showed that raters defined the task in three very different ways: (1) by searching the rubric to make a match between their response to the text and the language of the scoring rubric (search task definition), (2) by assigning a score directly based on a quick general impression (simple recognition task definition), or (3) by analyzing the criteria prior to score assignment without considering alternative scores (complex recognition task definition). Raters differed in their use of task definitions when they evaluated the same texts. These results challenged current Writing assessment procedures which assume that raters Internalize a scoring rubric during training and make a direct match between the scoring rubric and text characteristics. In addition, these results indicated that task definition is related to individual characteristics of the rater rather than status as a rater (i.e., external rater or 'teacher as rater'). These findings are discussed in terms of the effect of different task definitions on the validity of writing assessment.
| Identifer | oai:union.ndltd.org:LACETR/oai:collectionscanada.gc.ca:QMM.34716 |
| Date | January 1998 |
| Creators | DeRemer, Mary. |
| Contributors | Bracewell, Robert J. (advisor) |
| Publisher | McGill University |
| Source Sets | Library and Archives Canada ETDs Repository / Centre d'archives des thèses électroniques de Bibliothèque et Archives Canada |
| Language | English |
| Detected Language | English |
| Type | Electronic Thesis or Dissertation |
| Format | application/pdf |
| Coverage | Doctor of Philosophy (Department of Educational and Counselling Psychology.) |
| Rights | All items in eScholarship@McGill are protected by copyright with all rights reserved unless otherwise indicated. |
| Relation | alephsysno: 001615735, proquestno: NQ44407, Theses scanned by UMI/ProQuest. |
Page generated in 0.0018 seconds