Rater training (RT) improves the reliability of assessment tools, but has not been well studied for technical skills. This study assessed whether RT improved the psychometric properties of surgical skill assessments. Surgeons (N=47) were randomized to RT or non-training groups. The RT group underwent frame-of-reference training. Participants assessed trainees performing a suturing and knot-tying task using four assessment tools. Inter-rater reliability, initial and delayed rater agreement, and construct validity were assessed between groups. There was no significant effect of RT on the assessment tools’ reliability and validity. Reliability and validity were most robust for the global rating scale. Although there were trends towards improved reliability and validity with RT, confidence intervals were wide and overlapping. Reliability remained below the minimum desired level of 0.8 required for high-stakes testing. Although RT may represent a way to improve reliability, further study is needed to determine effective training methods. / February 2017

The Federal Aviation Administration (FAA) has been promoting Advanced Qualification Program (AQP) for pilot training and checking at Federal Aviation Regulations (FAR) Part 121 and Part 135 air carriers. Regarding pilot performance evaluation, instructors and evaluators assign scores to a student based on specific grading standards. To ensure the best possible quality of training and the highest level of safety, it is vital for different instructors and evaluators to grade students based on the same standard. Therefore, inter-rater and referent-rater agreement are paramount in calibrating the performance evaluation among different instructors and evaluators. This study was designed to test whether a focused workshop could increase the level of inter-rater and referent-rater agreement. A pre-test post-test control group experiment was conducted on a total of 29 Certified Flight Instructors (CFIs) at Purdue University. Participants were asked to watch several pre-scripted video flight scenarios recorded in an Embraer Phenom 100 FTD and give grades to the student pilots in the videos. After a rater training workshop that consisted of Behavior-Observation Training, Performance-Dimension Training, and Frame-of-Reference Training, participants in the treatment group were able to achieve a significantly higher level of inter-rater and referent-rater agreement.

Recently there has been increased interest in using employment interviews to judge personality (Christiansen, 2005; Powell & Goffin, 2009). However, not all personality traits are accurately detected. Previous research has demonstrated that Neuroticism is a trait that is difficult to correctly identify (Lippa & Dietz, 2000; Powell & Goffin, 2009; Watson, 1989). Neuroticism differs from the other Big Five traits because it is more affective in nature. It is proposed that those that are superior at detecting emotions in others will be better at judging Neuroticism in others. The purpose of the current research is to examine the relationship between emotional intelligence(specifically accuracy at detecting emotions) and accuracy in making personality judgments. A significant relation was found between emotional intelligence and accuracy in detecting Neuroticism. Implications for research and practice are discussed.

No description available.

Managerial self-awareness is thought to impact leadership. A multi-rater feedback instrument was used to gather performance data on 70 managers in a large multi-national airline in regards to five leadership dimensions: making sound decisions, driving for results, effective communication, self-management, and innovation. Difference scores between self and direct reports were calculated and used as the operational definition of managerial self-awareness. T-tests were run to examine the difference between high performers and average performers. No significant differences were found. Additionally, correlational measures between the five leadership competencies and the managerial self-awareness measure indicated statistically weak relationships.

Objective To identify inter-rater reliability and accuracy of pharmacy faculty members' classification of exam questions based on Bloom's Taxonomy. Methods Faculty at a college of pharmacy was given six example exam questions to assign to the appropriate Bloom's level. Results Inter-rater reliability and accuracy were both low at 0.25 and 46.0%, respectively. Accuracy increased to 81.8% when the six Bloom's levels collapsed to three. Conclusions Both inter-rater reliability and accuracy were low. Faculty members' misclassifications suggested a three-tier combination of the Bloom's levels that would optimally improve accuracy: Knowledge, Comprehension/Application, and Analysis/Synthesis/Evaluation. Faculty development should also be considered in improving accuracy and reliability.

Negative symptoms emerge early in the course of psychosis and are one of the best indicators of clinical outcomes; thus, the accurate, identification of these symptoms is paramount to early intervention efforts. Negative symptoms are typically assessed via self-report questionnaire or interviewer rated semi-structured interviews. However, the degree of concordance between self and interviewer report has been demonstrated to be low, approximately 50%. Therefore, it is important to identify the degree of discordance within each negative symptom domain (anhedonia, alogia, blunted affect, asociality, and amotivation) as well as to identify potential contributors to this lack of concordance. The present study examined each of the five negative symptom domains in regard to self and interviewer rating concordance in a sample (N = 70) that consists of a full range of negative symptoms and psychotic-like experiences (PLEs). Results indicated that self and interviewer report of negative symptom domains were inconsistently correlated. Dimensionally, asociality and avolition were the most concordant domains and alogia and anhedonia were more discordant. Although self and interviewer report scores of blunted affect did not significantly differ, each score may be capturing separate aspects of this experience. Participant drug use frequency was found to significantly impact rater responding on blunted affect, participant emotional clarity significantly impacted rater responding on alogia, participant clinical high risk (CHR) status impacted rater responding on consummatory anhedonia, and participant anxiety impacted rater responding on anticipatory anhedonia. This study identifies factors that underlie negative symptom rating discordance, which may help improve treatment intervention for those experiencing this critical phase of the disorder, and improve our ability to accurately monitor these symptoms in individuals with psychosis spectrum disorders. / Psychology

Elicited Imitation (EI), which is a way of assessing language learners' speaking, has been used for years. Furthermore, there have been many studies done showing rater bias (variance in test ratings associated with a specific rater and attributable to the attributes of a test taker) in language assessment. In this project, I evaluated possible rater bias, focusing mostly on bias attributable to raters' and test takers' language backgrounds, as seen in EI ratings. I reviewed literature on test rater bias, participated in a study of language background and rater bias, and produced recommendations for reducing bias in EI administration. Also, based on possible rater bias effects discussed in the literature I reviewed and on results of the research study I participated in, I created a registration tool to collect raters' background information that might be helpful in evaluating and reducing rater bias in future EI testing. My project also involved producing a co-authored research paper. In that paper we found no bias effect based on rater first or second language background.

The complexity and broadening of competencies have led to a number of assessment frameworks that advocate for the use of rater judgment in direct observation of clinical performance. The degree to which these assessment processes produce scores that are valid, are therefore vitally dependent on a rater’s cognitive ability. A number of theories suggest that many of the cognitive structures needed to complete rating tasks are capacity limited and may therefore become a source of difficulty when rating demands exceed resources. This thesis explores the role of rating demands on the performance and behaviour of raters in the assessment of clinical competence and asks: in what way do rating demands associated with rating clinical performance affect rater performance and behaviour? I hypothesized that as rating demands increase, rating performance declines and raters engage in cognitive avoidance strategies in order to complete the task. I tested this hypothesis by manipulating intrinsic and extraneous sources of load for raters in the assessment of clinical performance. Results consistently demonstrated that intrinsic load, specifically broadening raters’ focus by increasing the number of dimensions to be considered simultaneously, negatively affected indicators of rating quality. However, extraneous demands failed to result in the same effect in 2 of 3 experiments. When we explored the cognitive strategies raters engage under high load conditions we learned of a number of strategies to reduce cognitive work, including idiosyncratically minimizing intrinsic demands (leading to poor inter-rater reliability) and active elimination of sources of extraneous load, explaining both findings. When we induced extraneous load in manner that could not be easily minimized by raters, we also found impairments in rater performance, specifically the provision of feedback. I conclude that rating demands, whether induced intrinsically or by extraneous sources, impair rater performance affecting both the utility of scores and the opportunity for learner development. Implications for health professions education and future directions are discussed. / Dissertation / Doctor of Philosophy (PhD)

Book Summary: In the modern workplace, millions of dollars are spent each year on employee training. All too often, however, little thought and effort is spent on evaluating training and ensuring that the correct training is taking place. Successful training is about understanding both organizational goals and employee behavior. The basis for this book is the 9th Annual River Cities Industrial and Organizational Psychology Conference held on the campus of The University of Tennessee at Chattanooga in October 2013. The theme of the conference was Trends in Training. Accordingly, the focus of many of the talks was on how to train people to thrive and succeed in the environment where they spend a large portion of their lives the workplace.