The Federal Aviation Administration (FAA) has been promoting Advanced Qualification Program (AQP) for pilot training and checking at Federal Aviation Regulations (FAR) Part 121 and Part 135 air carriers. Regarding pilot performance evaluation, instructors and evaluators assign scores to a student based on specific grading standards. To ensure the best possible quality of training and the highest level of safety, it is vital for different instructors and evaluators to grade students based on the same standard. Therefore, inter-rater and referent-rater agreement are paramount in calibrating the performance evaluation among different instructors and evaluators. This study was designed to test whether a focused workshop could increase the level of inter-rater and referent-rater agreement. A pre-test post-test control group experiment was conducted on a total of 29 Certified Flight Instructors (CFIs) at Purdue University. Participants were asked to watch several pre-scripted video flight scenarios recorded in an Embraer Phenom 100 FTD and give grades to the student pilots in the videos. After a rater training workshop that consisted of Behavior-Observation Training, Performance-Dimension Training, and Frame-of-Reference Training, participants in the treatment group were able to achieve a significantly higher level of inter-rater and referent-rater agreement.

This replication study sought to analyze the effects of behavioral observability and overall impressions on rater agreement, as recently examined by Roch, Paquin, & Littlejohn (2009) and Scott (2012). Results from the study performed by Roch et al. indicated that raters are more likely to agree when items are either more difficult to rate or less observable. In the replication study conducted by Scott, the results did not support the relationship which Roch et al. found between observability and rater agreement, but did support the relationship previously found between item difficulty and rater agreement. The four objectives of this replication study were to determine whether rater agreement is negatively related to item observability (Hypothesis 1) and positively related to difficulty (Hypothesis 2), as well as to determine whether item performance ratings are closer to overall impressions when items are less observable (Hypothesis 3) and more difficult to rate (Hypothesis 4). The sample was comprised of 152 undergraduate students tasked with providing performance ratings on an individual depicted in a video of a discussion group. Results indicated that agreement was negatively correlated with both observability (supporting Hypothesis 1) and difficulty (not supporting Hypothesis 2), and that ratings were closer to overall impressions when items were less observable (supporting Hypothesis 3), but not when items were more difficult to rate (not supporting Hypothesis 4).

Significance tests for the measure of raw agreement are proposed. First, it is shown that the measure of raw agreement can be expressed as a proportionate reduction-in-error measure, sharing this characteristic with Cohen's Kappa and Brennan and Prediger's Kappa_n. Second, it is shown that the coefficient of raw agreement is linearly related to Brennan and Prediger's Kappa_n. Therefore, using the same base model for the estimation of expected cell frequencies as Brennan and Prediger's Kappa_n, one can devise significance tests for the measure of raw agreement. Two tests are proposed. The first uses Stouffer's Z, a probability pooler. The second test is the binomial test. A data example analyzes the agreement between two psychiatrists' diagnoses. The covariance structure of the agreement cells in a rater by rater table is described. Simulation studies show the performance and power functions of the test statistics. (author's abstract) / Series: Research Report Series / Department of Statistics and Mathematics

Multi-rater tools also referred to as 360-degree feedback tools, are frequently used in addition to traditional supervisory appraisals due to sources (i.e., supervisor, peer, direct report) unique perspectives and opportunities to view different aspects of job performance. Research has found that the differences among sources are most prevalent between self and other ratings, and the direction of agreement is related to overall job performance. Research has typically focused on one form of agreement, the direction of an individual's self-ratings compared to others' ratings. The current study expanded on past research on rater agreement using a data set (n = 215) consisting of multi-rater data for professionals participating in a leadership development process. The study examined the ability to predict job performance with three different measures of self-other agreement (i.e., difference between overall mean scores (difference), mean absolute difference across items (difference), and mean correlation across items (similarity)). The study also examined how the relationships may differ across performance dimensions. The final purpose was to explore how the importance of the performance dimensions, as rated by the participant, may moderate the relationship between self-other agreement and job performance. Partial support for study's hypotheses was found. The direction and difference measures of agreement on the overall multi-rater tool and performance dimensions accounted for a significant amount of the variance in job performance. The relationship between the similarity measure of agreement and job performance, and the moderating effect of importance were not supported in the current study.

No description available.

Cette thèse porte sur l’évaluation du leadership par une approche multi-évaluateurs (dite à 360 degrés). Tout d'abord les différents modèles du leadership, allant d'une conception unitaire, à une conception interractionniste puis à une définition en termes d'effet, sont détaillés. Puis en étudiant la question de la mesure, on met en évidence qu'avec la popularité croissante de leurs modèles, certains outils d'évaluation sont devenus des questionnaires psychométriques à part entière. Pourtant, du fait des limites de l’auto-évaluation, une nouvelle approche du leadership à vu le jour: l'évaluation à 360°. Elle évalue les qualités d'un manager en interrogeant les personnes travaillant avec lui (subordonnés, collègues, supérieurs...) et en comparant leurs évaluations avec la propre évaluation du manager. Un des points central de notre recherche a été d’étudier les apports et les limites de cette méthode. Tout d'abord on a vérifié dans quelle mesure les caractéristiques souvent utilisées pour expliquer la variabilité entre les catégories d'observateurs permettaient réellement de comprendre les écarts observés. Ces analyses ont montré que malgré des résultats significatifs, ces variables n’expliquent que très partiellement la variance existante. De par ces conclusions l’accent a été mis, non plus sur les différences inter-groupes, mais sur l'accord et le désaccord au sein des groupes d'observateurs. Enfin, la dernière partie revient aux bases méthodologiques et théoriques de la mesure en cherchant à proposer un modèle psychométriques qui conviendrait aux résultats de questionnaires à 360°, permettant de donner un cadre conceptuel au recours à des évaluateurs multiples. / This thesis is centered on leadership assessment through multi-rater evaluation, commonly known as 360 degrees assessment. First, leadership models were presented, and then, we discussed the measurement aspects of leadership, wherein we observed that some tools became full fledged psychometric assessments owning to the growing popularity of their underlying theory. Nevertheless, the concerns over the inherent limitations of self-report measures continued to be a major challenge in leadership assessment. This led to a new assessment approach called 360 degrees in which the characteristics of leaders are assessed by people working with them (subordinates, colleagues, superiors...) and compared with the leaders' self-appraisals. The focal point of this thesis was to study the advantages as well as the limitations of this approach. The study started with examining how well the variables which are supposed to explain the inter-rater variability were actually helpful in understanding the observed variance among observers. Those analysis yielded significant results despite the fact that those variables could explain only a very limited amount of variance. Following these observations, the study switched its focus from inter-group differences to intra-group / inter-rater agreement and disagreement. Finally, the last part of this thesis gets back to methodological and theoretical basics of measurement theory and proposes a psychometric model that would suit the 360 degrees assessments followed by a conceptual framework for the studies using multi-rater techniques.

<div> <div> <div> <p>Personality traits are important factors of psychotherapy for many reasons, as they relate to a variety of clinical outcomes, can complicate treatment, and can also be targets of treatment interventions. Because of its clinical prevalence and impact, it is imperative that therapists are able to effectively assess and treat personality pathology. Previous research has indicated that both client and therapist ratings of personality can provide meaningful information, and this varies across different sessions, but no study to date has examined both client and therapist ratings across the entire therapeutic intervention. There is also limited information on the agreement of client and therapist ratings of personality, as the majority of studies only examine the outset, the end, or a random time point of treatment. Examining only one point in time – or just the beginning and end – misses valuable information regarding possible changes in personality occurring throughout treatment. Using a naturalistic dataset of 128 client-therapist dyads (3,440 observations), the present study examined the longitudinal trajectory of client and therapist ratings of personality change throughout intervention while also accounting for state-level distress. Additionally, the agreement between clients and therapists were examined throughout treatment for any potential patterns of change using rank-order, mean-level, and absolute agreement. Significant patterns of trait change and change in absolute agreement across treatment were assessed using multilevel modeling. Last but not least, the agreement among clients and therapists were examined as potential predictors of therapeutic outcomes, such as engagement and improvement. The results provided evidence for significant decreases in neuroticism that were reported by the client but not therapist that suggest clients might be report decreases in state-level distress rather than true trait change. There were meaningful fluctuations in agreement across treatment, particularly for openness to experience and neuroticism, but the overall agreement – or lack thereof – did not significantly predict client engagement or improvement. Results highlight several clinical implications that are discussed. </p> </div> </div> </div>

Estimating others characteristics from facial cues plays an important role in our everyday lives. People usually agree in these estimates well and many of these estimates correlate. Majority studies consider one or few character- istics only and their respondents usually are in narrow ranges of ages. This study is partly based on rating of 13 characteristics of 80 men's and women's faces by respondents of various ages. These data were originally collected within yet unpublished study of Jaroslav Flegr, Amy E. Blum and Šebastian Kroupa. In this study I most strikingly found out that: i) older respondents of both genders rates photos of women as more attractive, ii) respondents spend more time by rating faces considered by themselves as more attractive or nice, iii) men rate people with different eye color as more attractive and women rate others with the same eye color as nicer, iv) preferences computed by two methods do not differ much. 1