This paper examined the inter-rater reliability of psychological assessments in practical field with 42 inmates’ PCL-R scores. As results, this study showed similar ICC and SEM values to those from PCL-manual. Concerning PCL-R structure, factor 2 showed higher ICC value than factor 1, and facet 4 showed higher ICC value than facet 1, 2, or 3. Especially, facet 2 showed low ICC value. Those are consistent with previous studies. However, ICC yielded by factor 2 only and both factor 1 and 2 showed similar ICC values. Considering theoretical and clinical aspects, it was recommendable to use PCL-R total score as risk assessment, though interpreting facet 2 requires cautions. Concerning to rater’s characteristics, the most influential factor to keep the PCL-R reliability was conducting it on regular basis, rather than licensed status. It was difficult to examine whether or not singed-off contribute to maintain sufficient reliability due to small sample size. In regression model, all rater related variables were not significantly correlated to PCL-R score change between two assessment occasions. PCL-R scores at Time 1 was moderately and negatively correlated to PCL-R score change. This indicated natural regression toward the mean. It is desirable to conduct additional study after obtaining more sample and rater related information, such as clinical experience. Additionally, it requires a consideration to apply findings in this study to female psychopathic subjects. As a policy implication, it is recommendable for personnel division to have psychologists to remain in their psychological work.

Magister Scientiae Dentium - MSc(Dent) / Aim: The aim of this study was to investigate the accuracy, intra- and inter-observer reliability of identifying occlusal markings made by articulating paper on complete dentures intra-orally. Methods: A series of photographs of 14 tissue borne complete dentures with occlusal markings was obtained. Articulating paper was used intra-orally at the delivery visit to make the occlusal markings. The denture sets were divided into two groups. Group 1 comprised pictures of the 14 complete lower dentures on their own, and group 2 comprised pictures of the same 14 lower dentures together with their opposing upper denture. The two groups of images were loaded into a Microsoft PowerPoint presentation as well as Keynote. Two experienced observers analysed the complete dentures independently and noted the number and distribution of the markings that they felt required adjustment. They differed, but discussed these and reached consensus. These data served as the control. Three groups of observers (10 per group) were then asked to analyse the occlusal markings of the 2 groups of denture images twice, with a two-week interval between each assessment. Before each subsequent assessment, the images were randomised by means of computer-generated random number sequence. The mean number of markings was established for each group and compared with the control mean. Intra-rater reliability was established by comparing the difference of the means of sequential observations for each rater by establishing the z-value. Inter-rater reliability within each group was established by means of analysis of variance. Results: Considering all the data, in only 17 instances (of the possible 60), did observers’ mean scores not differ from the control mean scores with good intra-rater reliability. In all other 43 instances the observers’ mean scores differed from the control mean scores and/or displayed poor intra-rater reliability. Considerable variation in inter-rater reliability was also found within every group of observers. Conclusion: The results indicate that observers are generally unable to reliably identify occlusal markings warranting occlusal adjustment, made by articulating paper on a lower complete denture. Clinical significance: Articulating paper should not be used intra-orally when delivering removable complete dentures.

No description available.

Objective To identify inter-rater reliability and accuracy of pharmacy faculty members' classification of exam questions based on Bloom's Taxonomy. Methods Faculty at a college of pharmacy was given six example exam questions to assign to the appropriate Bloom's level. Results Inter-rater reliability and accuracy were both low at 0.25 and 46.0%, respectively. Accuracy increased to 81.8% when the six Bloom's levels collapsed to three. Conclusions Both inter-rater reliability and accuracy were low. Faculty members' misclassifications suggested a three-tier combination of the Bloom's levels that would optimally improve accuracy: Knowledge, Comprehension/Application, and Analysis/Synthesis/Evaluation. Faculty development should also be considered in improving accuracy and reliability.

The purpose of this study was to determine the interrater reliability of the Texas Teacher Appraisal System instrument. The performance indicators, criteria, domains, and total instrument were analyzed for inter rater reliability. Five videotaped teaching episodes were viewed and scored by 557 to 881 school administrators trained to utilize the Texas Teacher Appraisal System. The fifty-five performance indicators were analyzed for simple percentage of agreement. The ten criteria, four performance domains, and) the whole instrument were analyzed utilizing Ruder-Richardson Formula 20. Indicators were judged reliable if there was 75 percent or greater agreement on four of the five videotaped exercises. Criteria, domains, and the whole instrument were judged reliable if they yielded a -Ruder-Richardson Formula 20 score of .75 or greater on four of the Based on the findings of this study, the following conclusions v/ere drawn: 1. Forty-eight of the fifty-five performance indicators were reliable in evaluation teacher performance. 2. Seven of the performance indicators were unreliable in evaluating teacher performance. 3. None of the ten performance criteria appeared to be reliable in evaluating teacher performance. 4. None of the four performance domains appeared to be reliable in evaluating teacher performance. 5. The whole instrument was reliable in evaluating teacher performance. 6. Reliability problems with the criteria and domains appeared to be an underestimate of reliability of the Kuder-Richardson Formula 20.

Data from rating scale assessments have rank-invariant properties only, which means that the data represent an ordering, but lack of standardized magnitude, inter-categorical distances, and linearity. Even though the judgments often are coded by natural numbers they are not really metric. The aim of this thesis is to further develop the nonparametric rank-based Svensson methods for paired ordinal data that are based on the rank-invariant properties only. The thesis consists of five papers. In Paper I the asymptotic properties of the measure of systematic disagreement in paired ordinal data, the Relative Position (RP), and the difference in RP between groups were studied. Based on the findings of asymptotic normality, two tests for analyses of change within group and between groups were proposed. In Paper II the asymptotic properties of rank-based measures, e.g. the Svensson’s measures of systematic disagreement and of additional individual variability were discussed, and a numerical method for approximation was suggested. In Paper III the asymptotic properties of the measures for paired ordinal data, discussed in Paper II, were verified by simulations. Furthermore, the Spearman rank-order correlation coefficient (rs) and the Svensson’s augmented rank-order agreement coefficient (ra) were compared. By demonstrating how they differ and why they differ, it is emphasized that they measure different things. In Paper IV the proposed test in Paper I for comparing two groups of systematic changes in paired ordinal data was compared with other nonparametric tests for group changes, both regarding different approaches of categorising changes. The simulation reveals that the proposed test works better for small and unbalanced samples. Paper V demonstrates that rank invariant approaches can also be used in analysis of ordinal data from multi-item scales, which is an appealing and appropriate alternative to calculating sum scores.

While each phase of the test development process is crucial to the validity of the examination, one phase tends to stand out among the others: the standard setting process. The standard setting process is a time-consuming and expensive endeavor. While it has received the most attention in the literature among any of the technical issues related to criterion-referenced measurement, little research attention has been given to generalizing the resulting performance standards. This procedure has the potential to improve the standard setting process by limiting the number of items rated and the number of individual rater decisions. The ability to generalize performance standards has profound implications both from a psychometric as well as a practicality standpoint. This study was conducted to evaluate the extent to which minimal competency estimates derived from a subset of multiple choice items using the Angoff standard setting method would generalize to the larger item set. Individual item-level estimates of minimal competency were simulated from existing and simulated item difficulty distributions. The study was designed to examine the characteristics of item sets and the standard setting process that could impact the ability to generalize a single performance standard. The characteristics and the relationship between the two item sets included three factors: (a) the item difficulty distributions, (b) the location of the 'true' performance standard, (c) the number of items randomly drawn in the sample. The characteristics of the standard setting process included four factors: (d) number of raters, (e) percentage of unreliable raters, (f) magnitude of 'unreliability' in unreliable raters, and (g) the directional influence of group dynamics and discussion. The aggregated simulation results were evaluated in terms of the location (bias) and the variability (mean absolute deviation, root mean square error) in the estimates. The simulation results suggest that the model of using partial item sets may have some merit as the resulting performance standard estimates may 'adequately' generalize to those set with larger item sets. The simulation results also suggest that elements such as the distribution of item difficulty parameters and the potential for directional group influence may also impact the ability to generalize performance standards and should be carefully considered.

<p>Syftet med denna studie var att undersöka reliabiliteten i en metod för att observera affektuttryck, Stålforsmetoden. Stålforsmetoden fokuserar primärt på affektuttryck i ansiktet, och mer specifikt den första affekten som en patient uttrycker under en psykoterapisession (”överföringsaffekt”). Den teoretiska grunden är affektteori som utvecklats av Silvan Tomkins och Paul Ekman. Data har samlats in med strukturerad observation och analyseras kvantitativt. Interbedömarreliabilitet beräknades med Cohen’s Kappa och uppgick till K = 0,03, vilket innebär att det inte finns någon statistiskt säker överensstämmelse mellan bedömarna. Möjliga orsaker till avsaknaden av interbedömarreliabilitet såsom utbildningens utformning och omfattning samt operationaliseringen av observationsvariabeln diskuteras och förslag på framtida forskning lämnas.</p> / <p>The purpose of this study was to explore the reliability of a method for observing expressions of affect, “Stålforsmetoden”. Stålforsmetoden focuses primarily on facial expression of affect, and more precisely the first expression presented by a patient in a psychotherapy session (referred to as transference affect). The theoretical basis is affect theory as developed in the works of Silvan Tomkins and Paul Ekman respectively. Data has been collected through structured observation and analyzed quantitatively. Inter-rater reliability was calculated using Cohen’s Kappa and amounted to K = 0.03, which means that there was no significant agreement between raters. This result implies that the reliability of Stålforsmetoden in its present form is insufficient and that further development of the method is needed. Possible reasons for the absence of inter-rater reliability such as the adequacy of education and the operationalization of transference affect are discussed and suggestions for future research are presented.</p>

Syftet med denna studie var att undersöka reliabiliteten i en metod för att observera affektuttryck, Stålforsmetoden. Stålforsmetoden fokuserar primärt på affektuttryck i ansiktet, och mer specifikt den första affekten som en patient uttrycker under en psykoterapisession (”överföringsaffekt”). Den teoretiska grunden är affektteori som utvecklats av Silvan Tomkins och Paul Ekman. Data har samlats in med strukturerad observation och analyseras kvantitativt. Interbedömarreliabilitet beräknades med Cohen’s Kappa och uppgick till K = 0,03, vilket innebär att det inte finns någon statistiskt säker överensstämmelse mellan bedömarna. Möjliga orsaker till avsaknaden av interbedömarreliabilitet såsom utbildningens utformning och omfattning samt operationaliseringen av observationsvariabeln diskuteras och förslag på framtida forskning lämnas. / The purpose of this study was to explore the reliability of a method for observing expressions of affect, “Stålforsmetoden”. Stålforsmetoden focuses primarily on facial expression of affect, and more precisely the first expression presented by a patient in a psychotherapy session (referred to as transference affect). The theoretical basis is affect theory as developed in the works of Silvan Tomkins and Paul Ekman respectively. Data has been collected through structured observation and analyzed quantitatively. Inter-rater reliability was calculated using Cohen’s Kappa and amounted to K = 0.03, which means that there was no significant agreement between raters. This result implies that the reliability of Stålforsmetoden in its present form is insufficient and that further development of the method is needed. Possible reasons for the absence of inter-rater reliability such as the adequacy of education and the operationalization of transference affect are discussed and suggestions for future research are presented.

As ultrasound imaging gains popularity in speech research, an important question to address is the reliability of the measures taken from these images. This study examines the reliability of hand measures of ultrasound data collected by graduate student researchers in the University of South Florida's speech science lab. Speech production data from Ultrasound analysis of velar fronting (Wodzinski, 2004) and Ultrasound study of errors in speech production (Frisch, 2003) were used to obtain inter-rater reliability measures. This study compares the raters choice of video frame depicting alveolar or velar closure image, anterior and posterior points of closure, tongue blade and velar angle measurements, as well as a measurement of the tongue dorsum distance from the ultrasound probe.