Multi-rater tools also referred to as 360-degree feedback tools, are frequently used in addition to traditional supervisory appraisals due to sources (i.e., supervisor, peer, direct report) unique perspectives and opportunities to view different aspects of job performance. Research has found that the differences among sources are most prevalent between self and other ratings, and the direction of agreement is related to overall job performance. Research has typically focused on one form of agreement, the direction of an individual's self-ratings compared to others' ratings. The current study expanded on past research on rater agreement using a data set (n = 215) consisting of multi-rater data for professionals participating in a leadership development process. The study examined the ability to predict job performance with three different measures of self-other agreement (i.e., difference between overall mean scores (difference), mean absolute difference across items (difference), and mean correlation across items (similarity)). The study also examined how the relationships may differ across performance dimensions. The final purpose was to explore how the importance of the performance dimensions, as rated by the participant, may moderate the relationship between self-other agreement and job performance. Partial support for study's hypotheses was found. The direction and difference measures of agreement on the overall multi-rater tool and performance dimensions accounted for a significant amount of the variance in job performance. The relationship between the similarity measure of agreement and job performance, and the moderating effect of importance were not supported in the current study.

Problem Statement: Because of the lack of evidence in support of a pathology-based approach to low back pain (LBP), many researchers advocate the use of classification systems that classify patients with LBP based on common clinical characteristics. Treatment based on direction tolerance can improve circulation, diminish pain and inflammation, restore normal motion, and reduce fear-avoidance beliefs. Treatment performed in the standing position can functionally integrate related regions of the body, can replicate normal tissue loading in standing position, replicate normal proprioceptive demands of motion performed in standing position, and address underlying movement impairments in related regions that may contribute to compensatory tissue pathology in the low back. The problem is that none of the existing functional classification systems for evaluation and treatment of LBP are designed for this purpose. The Direction Tolerance System (DTS) is a new functional classification system that is based on direction tolerance and performed entirely in the standing position. Objectives: The purpose of this research was to investigate the reliability, outcome, and prognostic ability of the DTS for evaluation and treatment of LBP. Methods: This exploratory research was a prospective cohort study with 60 patients seeking physical therapy (PT) for treatment of LBP by 5 different therapists. All patients were evaluated and treated for 8 visits using the DTS. Reliability was measured by comparing DTS scores (DTI) on visits 1 and 2. DTS treatment consisted of a scripted flexibility and strengthening exercise program based on motions that were determined to be non-aggravating during the DTS examination. Outcome was assessed using the DTI, modified Oswestry Disability Questionnaire (ODQ), Numeric Pain Rating Scale (NPRS) and Fear-Avoidance Beliefs Questionnaire Physical Activity Subscale (PABQ-PA). Baseline variables were assessed for correlation with successful outcome and ability to predict a successful outcome. Results: (1) Group intra-rater reliability between visits 1 and 2 using ICC (3,1) was good (.80, p<.001); (2) The group success rate was 62%, which exceeded the expected 50% success rate. (3) Baseline variables were not associated with a successful outcome and did not significantly predict a successful outcome. Discussion: The DTS is a simple, reliable and effective method of evaluating and treating LBP, and with proper training, can be used by clinicians.

Background: Muscle thickness, pennation angle, and fascicle length describe the architecture of a muscle. These properties can be observed alongside subcutaneous fat thickness using ultrasonography; however, measurement is sensitive to the angle of the transducer against the skin. Typically, the transducer is held perpendicular to skin for imaging. Nonetheless, a convenient, reliable method to ensure transducer angle consistency has not been reported. Objectives: The purpose of this study was to determine the influence of transducer angle on muscle architecture and subcutaneous fat measurements of quadriceps muscles (vastus lateralis and rectus femoris) in healthy young adults. A secondary objective was to determine intra- and inter-rater reliability. Methods: Thirty men and women were recruited (25±2.5 years; BMI: 22.6±3.0 kg/m2). Ultrasound images were acquired from two muscles. An image was taken at an estimated perpendicular angle to the skin. Then, using a 3D-printed device with a protractor that attached to the ultrasound transducer, images were taken at measured angles 5-10˚ medial and lateral to perpendicular. Agreement and error were determined using intraclass correlation coefficients (ICCs) and standard error measurements (SEMs). Results: Good to excellent agreement was demonstrated for muscle and fat thicknesses regardless of transducer angle (ICC >0.66). Intra-rater reliability was excellent for all outcomes within both muscles (ICC >0.89). Inter-rater reliability for the rectus femoris was good to excellent for all transducer angles except for measurements of fascicle lengths at 85° (ICC: 0.33–0.99). Inter-rater reliability improved >20% for the vastus lateralis with the device. Conclusion: Measurements of muscle pennation angle and fascicle lengths, but not muscle or subcutaneous fat thicknesses, were sensitive to transducer angle. Reliability of pennation angle and fascicle lengths improved with the use of our device. Using our device, reliable muscle architecture measures can be made for the rectus femoris and the vastus lateralis in healthy young adults. / Thesis / Master of Science (MSc) / The arrangement of small muscle components, known as fascicles, can be observed in humans using ultrasound imaging. These fascicle arrangements can be measured to improve understanding of muscle function and disease processes. A potential problem of viewing muscle with ultrasound is that the angle of the probe head against the skin can alter the appearance of the muscle fascicles. The goal of this research was to improve current methods of ultrasound imaging of two thigh muscles. We have created a novel 3D-printed device to attach to the existing ultrasound probe. This 3D-printed device stabilizes the ultrasound probe head; and accurately determines the angles between the ultrasound probe head and the surface of the skin. In this study, the use of this device improved reliability of our ultrasound images by >20%. Future use of this device may improve measurements of muscle fascicles acquired with ultrasound imaging.

Rater training is important for various human resource activities. The proposed symposium highlights contemporary research on rater training in different arenas such as performance appraisal, job interviews, and foreign language testing. New and emerging trends in rater training research and practice will also be discussed.

Despite Landy and Farr’s (1980) highly influential call for a moratorium on rating format research over 30 years ago, recent research continues to show that rating format design can influence rating quality. This symposium brings together a group of scholars who will share their research and expertise on this topic.

No description available.

No description available.

This study investigated how 2 different rating conditions, the controlled rating condition (CRC) and the uncontrolled rating condition (URC), effected rater behavior and the reliability of a performance assessment (PA) known as the Missionary Teaching Assessment (MTA). The CRC gives raters the capability to manipulate (pause, rewind, fast-forward) video recordings of an examinee's performance as they rate while the URC does not give them this capability (i.e., the rater must watch the recording straight through without making any manipulations). Few studies have compared the effect of these two rating conditions on ratings. Ryan et al. (1995) analyzed the impact of the CRC and URC on the accuracy of ratings, but few, if any, have analyzed its impact on reliability. The Missionary Teaching Assessment is a performance assessment used to assess the teaching abilities of missionaries for the Church of Jesus Christ of Latter-day Saints at the Missionary Training Center. In this study, 32 missionaries taught a 10-minute lesson that was recorded and later rated by trained raters based on a rubric containing 5 criteria. Each teaching sample was rated by 4 of 6 raters. Two of the 4 ratings were rated using the CRC and 2 using the URC. Camtasia Studio (2010), a screen capture software, was used to record when raters used any type of manipulation. The recordings were used to analyze if raters manipulated the recordings and if so, when and how frequently. Raters also performed think-alouds following a random sample of the ratings that were performed using the CRC. These data revealed that when raters had access to the CRC they took advantage of it the majority of the time, but they differed in how frequently they manipulated the recordings. The CRC did not add an exorbitant amount of time to the rating process. The reliability of the ratings was analyzed using both generalizability theory (G theory) and many-facets Rasch measurement (MFRM). Results indicated that, in general, the reliability of the ratings obtained from the 2 rating conditions were not statistically significantly different from each other. The implications of these findings are addressed.

In the research regarding rating ESL writing assessments, borderline writing samples are mentioned, but a solution has yet to be addressed. Borderline samples are writing samples that do not perfectly fit a set level within the rubric, but instead have characteristics from multiple levels. The aim of this thesis is to provide an improved training module in the setting of an Intensive English Program by exposing new raters to borderline samples and rating rationale from experienced raters. The purpose of this training is to increase the confidence, consistency, and accuracy of novice raters when rating borderline samples of writing. The training consists of a workbook with a rubric and instructions for use, benchmark examples of writing, borderline examples of writing with comments from experienced raters defending the established scores, then a variety of writing samples for practice. The selection of the benchmark and the borderline examples of writing was informed by the fit statistic from existing datasets that had been analyzed with many-facet Rasch measurement. Eight experienced raters participated in providing rationale based upon the rubric explaining why each borderline sample was rated with its established score, and describing why the sample could be considered at a different level. In order to assess the effectiveness of the training workbook, it was piloted by 10 novice raters who rated a series of essays and responded to a survey. Results of the survey demonstrated that rater confidence increased following the training, but that they needed more time with the training materials to use them properly. The statistical analyses showed insignificant changes, which could be due to the limitations of the data collection. Further research regarding the effectiveness of this training workbook is necessary, as well as an increased discussion in the field regarding the prevalent issue of rating borderline samples of writing.

In spite of growing scholarship on environmentally induced non-migration research in Africa, comprehensive empirical evidence of non-migration drivers is extremely difficult to find. We review 77 rigorously selected empirical articles on the drivers of environmental non-migration. A variety of relevant keywords was applied to search, identify, and select key publications from ScienceDirect, Web of Knowledge, Google Scholar, and the Climig databases. Content analysis and inter-rater reliability (IRR) analysis were used to summarize the literature and identify key drivers of environmental non-migration decisions across all retained articles. The study structure was informed by the Foresight (2011a) conceptual framework. A growth in the non-migration literature across the time period was observed. Social factors, particularly place-based attachment and family/cultural obligations, was identified as the most important driver of non-migration (IRR score = 0.67). Environmental factors were ranked second, particularly the ability of the affected to develop coping capacity through experiential learning even in contexts marred by resource scarcity and widespread poverty. Given the limited literature on environmental nonmigration decisions, we recommend increased non-migration research across Africa to better inform policy decisions. This is particularly important as climate-related disasters surge. Frequent reviews on diverse aspects of non-migration studies are recommended to redefine future research and non-migration policy considerations in Africa.