In the research regarding rating ESL writing assessments, borderline writing samples are mentioned, but a solution has yet to be addressed. Borderline samples are writing samples that do not perfectly fit a set level within the rubric, but instead have characteristics from multiple levels. The aim of this thesis is to provide an improved training module in the setting of an Intensive English Program by exposing new raters to borderline samples and rating rationale from experienced raters. The purpose of this training is to increase the confidence, consistency, and accuracy of novice raters when rating borderline samples of writing. The training consists of a workbook with a rubric and instructions for use, benchmark examples of writing, borderline examples of writing with comments from experienced raters defending the established scores, then a variety of writing samples for practice. The selection of the benchmark and the borderline examples of writing was informed by the fit statistic from existing datasets that had been analyzed with many-facet Rasch measurement. Eight experienced raters participated in providing rationale based upon the rubric explaining why each borderline sample was rated with its established score, and describing why the sample could be considered at a different level. In order to assess the effectiveness of the training workbook, it was piloted by 10 novice raters who rated a series of essays and responded to a survey. Results of the survey demonstrated that rater confidence increased following the training, but that they needed more time with the training materials to use them properly. The statistical analyses showed insignificant changes, which could be due to the limitations of the data collection. Further research regarding the effectiveness of this training workbook is necessary, as well as an increased discussion in the field regarding the prevalent issue of rating borderline samples of writing.
| Identifer | oai:union.ndltd.org:BGMYU2/oai:scholarsarchive.byu.edu:etd-5065 |
| Date | 16 June 2013 |
| Creators | Greer, Brittney |
| Publisher | BYU ScholarsArchive |
| Source Sets | Brigham Young University |
| Detected Language | English |
| Type | text |
| Format | application/pdf |
| Source | Theses and Dissertations |
| Rights | http://lib.byu.edu/about/copyright/ |
Page generated in 0.0025 seconds