This replication study sought to analyze the effects of behavioral observability and overall impressions on rater agreement, as recently examined by Roch, Paquin, & Littlejohn (2009) and Scott (2012). Results from the study performed by Roch et al. indicated that raters are more likely to agree when items are either more difficult to rate or less observable. In the replication study conducted by Scott, the results did not support the relationship which Roch et al. found between observability and rater agreement, but did support the relationship previously found between item difficulty and rater agreement. The four objectives of this replication study were to determine whether rater agreement is negatively related to item observability (Hypothesis 1) and positively related to difficulty (Hypothesis 2), as well as to determine whether item performance ratings are closer to overall impressions when items are less observable (Hypothesis 3) and more difficult to rate (Hypothesis 4). The sample was comprised of 152 undergraduate students tasked with providing performance ratings on an individual depicted in a video of a discussion group. Results indicated that agreement was negatively correlated with both observability (supporting Hypothesis 1) and difficulty (not supporting Hypothesis 2), and that ratings were closer to overall impressions when items were less observable (supporting Hypothesis 3), but not when items were more difficult to rate (not supporting Hypothesis 4).

No description available.

No description available.