This study examined two item characteristics believed to influence rater agreement: observability and difficulty. The first goal of this study was to replicate the findings of Roch, Paquin and Littlejohn (2009), which found that rater agreement was negatively related to item observability (Hypothesis 1) and rating difficulty (Hypothesis 2). The study also explored whether participants had closer item performance ratings to their overall impression when items were less observable (Hypothesis 3) and more difficult to rate (Hypothesis 4). A sample of 254 Undergraduate psychology students viewed a video of a leaderless group discussion and then filled out a rating form assessing performance of one of the individuals in the video and rating difficulty. Results were that rater agreement was positively related to observability (not supporting Hypothesis 1) and negatively related to difficulty (supporting Hypothesis 2). RDS, a distance score between participant’s overall impression and the item performance rating was computed to assess Hypotheses 3 and 4. RDS was positively related to observability (supporting Hypothesis 3) and not related to difficulty (not supporting Hypothesis 4). The positive relationship between observability and rater agreement was surprising given that it was the opposite of previous findings. Not hypothesized but of interest to the study was that observability and difficulty were not correlated. In previous studies, these variables were negatively correlated. Implications of these findings are discussed along with directions for further research.
| Identifer | oai:union.ndltd.org:WKU/oai:digitalcommons.wku.edu:theses-2155 |
| Date | 01 May 2012 |
| Creators | Scott, Jennifer N. |
| Publisher | TopSCHOLAR® |
| Source Sets | Western Kentucky University Theses |
| Detected Language | English |
| Type | text |
| Format | application/pdf |
| Source | Masters Theses & Specialist Projects |
Page generated in 0.0015 seconds