Prior investigation of a new rater training paradigm, rater variability training (RVT), found no clear empirical distinction between RVT and the more established frame-of-reference training (FOR), (Hauenstein, Facteau, & Schmidt, 1999). The purpose of the present study was to expand upon this previous investigation by including a purpose manipulation, alternative operationalizations of Cronbach's accuracy components, finer-grained distinctions in the rating stimuli, and a second control group receiving quantitative accuracy feedback void of a substantive training lecture. Results indicate that finer-grained distinctions in the rating stimuli result in the best differential elevation accuracy for RVT trainees. Furthermore, RVT may be best suited for improving raters' abilities to accurately evaluate average performing ratees when the performance appraisal is used for an administrative purpose. Evidence also suggests that in many cases, the use of Cronbach's accuracy components obscures underlying patterns of rating accuracy. Finally, there is evidence to suggest that accuracy feedback without a training lecture improves some types of rating accuracy. / Master of Science

Rater training (RT) improves the reliability of assessment tools, but has not been well studied for technical skills. This study assessed whether RT improved the psychometric properties of surgical skill assessments. Surgeons (N=47) were randomized to RT or non-training groups. The RT group underwent frame-of-reference training. Participants assessed trainees performing a suturing and knot-tying task using four assessment tools. Inter-rater reliability, initial and delayed rater agreement, and construct validity were assessed between groups. There was no significant effect of RT on the assessment tools’ reliability and validity. Reliability and validity were most robust for the global rating scale. Although there were trends towards improved reliability and validity with RT, confidence intervals were wide and overlapping. Reliability remained below the minimum desired level of 0.8 required for high-stakes testing. Although RT may represent a way to improve reliability, further study is needed to determine effective training methods. / February 2017

Book Summary: In the modern workplace, millions of dollars are spent each year on employee training. All too often, however, little thought and effort is spent on evaluating training and ensuring that the correct training is taking place. Successful training is about understanding both organizational goals and employee behavior. The basis for this book is the 9th Annual River Cities Industrial and Organizational Psychology Conference held on the campus of The University of Tennessee at Chattanooga in October 2013. The theme of the conference was Trends in Training. Accordingly, the focus of many of the talks was on how to train people to thrive and succeed in the environment where they spend a large portion of their lives the workplace.

Rater training is important for various human resource activities. The proposed symposium highlights contemporary research on rater training in different arenas such as performance appraisal, job interviews, and foreign language testing. New and emerging trends in rater training research and practice will also be discussed.

Despite Landy and Farr’s (1980) highly influential call for a moratorium on rating format research over 30 years ago, recent research continues to show that rating format design can influence rating quality. This symposium brings together a group of scholars who will share their research and expertise on this topic.

No description available.

No description available.

In the research regarding rating ESL writing assessments, borderline writing samples are mentioned, but a solution has yet to be addressed. Borderline samples are writing samples that do not perfectly fit a set level within the rubric, but instead have characteristics from multiple levels. The aim of this thesis is to provide an improved training module in the setting of an Intensive English Program by exposing new raters to borderline samples and rating rationale from experienced raters. The purpose of this training is to increase the confidence, consistency, and accuracy of novice raters when rating borderline samples of writing. The training consists of a workbook with a rubric and instructions for use, benchmark examples of writing, borderline examples of writing with comments from experienced raters defending the established scores, then a variety of writing samples for practice. The selection of the benchmark and the borderline examples of writing was informed by the fit statistic from existing datasets that had been analyzed with many-facet Rasch measurement. Eight experienced raters participated in providing rationale based upon the rubric explaining why each borderline sample was rated with its established score, and describing why the sample could be considered at a different level. In order to assess the effectiveness of the training workbook, it was piloted by 10 novice raters who rated a series of essays and responded to a survey. Results of the survey demonstrated that rater confidence increased following the training, but that they needed more time with the training materials to use them properly. The statistical analyses showed insignificant changes, which could be due to the limitations of the data collection. Further research regarding the effectiveness of this training workbook is necessary, as well as an increased discussion in the field regarding the prevalent issue of rating borderline samples of writing.