<p>Syftet med denna studie var att undersöka reliabiliteten i en metod för att observera affektuttryck, Stålforsmetoden. Stålforsmetoden fokuserar primärt på affektuttryck i ansiktet, och mer specifikt den första affekten som en patient uttrycker under en psykoterapisession (”överföringsaffekt”). Den teoretiska grunden är affektteori som utvecklats av Silvan Tomkins och Paul Ekman. Data har samlats in med strukturerad observation och analyseras kvantitativt. Interbedömarreliabilitet beräknades med Cohen’s Kappa och uppgick till K = 0,03, vilket innebär att det inte finns någon statistiskt säker överensstämmelse mellan bedömarna. Möjliga orsaker till avsaknaden av interbedömarreliabilitet såsom utbildningens utformning och omfattning samt operationaliseringen av observationsvariabeln diskuteras och förslag på framtida forskning lämnas.</p> / <p>The purpose of this study was to explore the reliability of a method for observing expressions of affect, “Stålforsmetoden”. Stålforsmetoden focuses primarily on facial expression of affect, and more precisely the first expression presented by a patient in a psychotherapy session (referred to as transference affect). The theoretical basis is affect theory as developed in the works of Silvan Tomkins and Paul Ekman respectively. Data has been collected through structured observation and analyzed quantitatively. Inter-rater reliability was calculated using Cohen’s Kappa and amounted to K = 0.03, which means that there was no significant agreement between raters. This result implies that the reliability of Stålforsmetoden in its present form is insufficient and that further development of the method is needed. Possible reasons for the absence of inter-rater reliability such as the adequacy of education and the operationalization of transference affect are discussed and suggestions for future research are presented.</p>

Syftet med denna studie var att undersöka reliabiliteten i en metod för att observera affektuttryck, Stålforsmetoden. Stålforsmetoden fokuserar primärt på affektuttryck i ansiktet, och mer specifikt den första affekten som en patient uttrycker under en psykoterapisession (”överföringsaffekt”). Den teoretiska grunden är affektteori som utvecklats av Silvan Tomkins och Paul Ekman. Data har samlats in med strukturerad observation och analyseras kvantitativt. Interbedömarreliabilitet beräknades med Cohen’s Kappa och uppgick till K = 0,03, vilket innebär att det inte finns någon statistiskt säker överensstämmelse mellan bedömarna. Möjliga orsaker till avsaknaden av interbedömarreliabilitet såsom utbildningens utformning och omfattning samt operationaliseringen av observationsvariabeln diskuteras och förslag på framtida forskning lämnas. / The purpose of this study was to explore the reliability of a method for observing expressions of affect, “Stålforsmetoden”. Stålforsmetoden focuses primarily on facial expression of affect, and more precisely the first expression presented by a patient in a psychotherapy session (referred to as transference affect). The theoretical basis is affect theory as developed in the works of Silvan Tomkins and Paul Ekman respectively. Data has been collected through structured observation and analyzed quantitatively. Inter-rater reliability was calculated using Cohen’s Kappa and amounted to K = 0.03, which means that there was no significant agreement between raters. This result implies that the reliability of Stålforsmetoden in its present form is insufficient and that further development of the method is needed. Possible reasons for the absence of inter-rater reliability such as the adequacy of education and the operationalization of transference affect are discussed and suggestions for future research are presented.

As ultrasound imaging gains popularity in speech research, an important question to address is the reliability of the measures taken from these images. This study examines the reliability of hand measures of ultrasound data collected by graduate student researchers in the University of South Florida's speech science lab. Speech production data from Ultrasound analysis of velar fronting (Wodzinski, 2004) and Ultrasound study of errors in speech production (Frisch, 2003) were used to obtain inter-rater reliability measures. This study compares the raters choice of video frame depicting alveolar or velar closure image, anterior and posterior points of closure, tongue blade and velar angle measurements, as well as a measurement of the tongue dorsum distance from the ultrasound probe.

A predictive validity approach has been employed to find some implications to support evidences for Automated Essay Scoring (AES) systems. First, using R² values from multiple linear regression models, validity indices are compared first between multiple choice scores and essay scores across four AES systems. Secondly, R² values from models using only essay scores, the validity indices of four AES systems are hypothetically compared to see if how well AES systems could predict student outcome such as GPA. / text

Setting performance standards is a judgmental process involving human opinions and values as well as technical and empirical considerations and although all cut score decisions are by nature arbitrary, they should not be capricious. Establishing a minimum passing standard is the technical expression of a policy decision and the information gained through standard setting studies inform these policy decisions. To this end, it is necessary to conduct robust examinations of methods and techniques commonly applied to standard setting studies in order to better understand issues that may influence policy decisions. The modified-Angoff method remains one of the most popular methods for setting performance standards in testing and assessment. With this method, is common practice to provide content experts with feedback regarding the item difficulties; however, it is unclear how this feedback affects the ratings and recommendations of content experts. Recent research seems to indicate mixed results, noting that the feedback given to raters may or may not alter their judgments depending on the type of data provided, when the data was provided, and how raters collaborated within groups and between groups. This research seeks to examine issues related to the effects of item-level feedback on the judgment of raters. The results suggest that the most important factor related to item-level feedback is whether or not a Subject Matter Expert (SME) was able to correctly answer a question. If so, then the SMEs tended to rely on their own inherent sense of item difficulty rather than the data provided, in spite of empirical evidence to the contrary. The results of this research may hold implications for how standard setting studies are conducted with regard to the difficulty and ordering of items, the ability level of content experts invited to participate in these studies, and the types of feedback provided.

Significance tests for the measure of raw agreement are proposed. First, it is shown that the measure of raw agreement can be expressed as a proportionate reduction-in-error measure, sharing this characteristic with Cohen's Kappa and Brennan and Prediger's Kappa_n. Second, it is shown that the coefficient of raw agreement is linearly related to Brennan and Prediger's Kappa_n. Therefore, using the same base model for the estimation of expected cell frequencies as Brennan and Prediger's Kappa_n, one can devise significance tests for the measure of raw agreement. Two tests are proposed. The first uses Stouffer's Z, a probability pooler. The second test is the binomial test. A data example analyzes the agreement between two psychiatrists' diagnoses. The covariance structure of the agreement cells in a rater by rater table is described. Simulation studies show the performance and power functions of the test statistics. (author's abstract) / Series: Research Report Series / Department of Statistics and Mathematics

Legislative pressures are being brought to bear on South African employers to demonstrate that occupational assessment is scientifically valid and culturefair. The development of valid and reliable performance-based assessment tools will enable employers to meet these requirements. The general aim of this research was to validate a performance-based assessment battery for the placement of sales representatives. A literature survey examined alternative assessment measures and methods of performance measurement, leading to the conclusion that the combination of the work sample as a predictor measure and the managerial rating of performance as a criterion measure offer a practical and cost-effective assessment process to the sales manager. The empirical study involved 54 sales persons working for the Commercial division of an oil marketing company, selling products and services to the commercial and industrial market. By means of the empirical study, a significant correlation was found between performance of sales representatives in terms of the performance-based assessment battery for the entry level of the career ladder and their behaviour in the field as measured by the managerial performance rating instrument. The limitations of the sample, however, prevent the results from being generalised to other organisations.

Unga med normbrytande beteende löper en relativt hög risk för en långvarig negativ utveckling. För att förhindra detta krävs tidiga effektiva insatser som i sin tur kräver tillförlitliga bedömningsinstrument som identifierar risker och behov hos unga med, eller i riskzonen för normbrytande beteende. Just detta är syftet med ESTER-bedömning. Föreliggande studies syfte var att undersöka interbedömarreliabiliteten av ESTER-bedömning inklusive en ny kandidatskala för riskfaktorerna . Två oberoende bedömare genomförde ESTER-bedömningar på journalmaterial tillhörande 30 tvångsomhändertagna flickor, 15-20 år. Resultaten visar en spridning mellan bristfällig till mycket bra interbedömarreliabilitet på de 19 risk- och skyddsfaktorerna i ESTER-bedömning, med få fall av total oenighet mellan bedömarna. En jämförelse mellan den befintliga skalan och kandidatskalan visade marginella skillnader. Vidare forskning av interbedömarreliabilitet för ESTER-bedömning bör testa skalorna var för sig och inkludera intervjuer som informationskälla. / Youths with normbreaking behavior is at higher risk for a negative development. To prevent this, there is a need for reliable assessments that can identify risk and need for youths with, or at risk for normbreaking behavior. This is the purpose of ESTER-assessment. This study evaluated the inter-rater reliability of two different scales in ESTER-assessment. Two independent judges conducted ESTER-assessment on case files of 30 institutionalized girls, aged 15-20 years. The results revealed poor to excellent agreement and few cases of total disagreements. The two different scales showed a minimal difference. In further research of the inter-rater reliability of ESTER-assessment there is a need for testing the two scales separately and to include interviews as a source of information.

Children with Attention Deficit Hyperactivity Disorder (ADHD) tend to experience social functioning problems, with girls more likely to encounter peer rejection than boys. The present study investigated gender differences in child, parent, and teacher perceptions of social functioning among ADHD and control children. Participants included 119 children (ages 6-11) and their parents. Sixty-one children were previously diagnosed with ADHD. Parents, teachers, and children completed measures assessing the child's social functioning. The results indicate that the relationship between ADHD status and social functioning differs as a function of rater. Teachers and parents reported that ADHD children had lower social functioning than controls, while ADHD and control children reported similar levels of social functioning. Gender differences were found on the child self-report, with girls reporting lower social functioning than boys. In ADHD children the relationship between social functioning and comorbid depression differed as a function of rater. Specifically, among ADHD children with depression, parents rated children as having lower social functioning than did children or teachers. In ADHD children without comorbid depression, however, there were no rater differences. Additionally, no rater differences in social functioning were found between ADHD children with and without a comorbid psychiatric condition. Overall, the results of the current study lend support to the idea that parents, teachers, and children have different perceptions of social functioning. Clinically, these results suggest that interventions could focus on identifying those ADHD children most at-risk for social functioning problems and developing interventions that fit with their perceptions. The limitations of the current study and directions for future research are presented.

<div>This dissertation investigates three utterance fluency features and two vocabulary features of 409 speech samples from advanced intermediate and advanced L2 English speakers, who participated in the Oral English Proficiency Test (OEPT) between the year of 2009 and 2015. Among the 409 L2 English speakers, there are 80 L1 Hindi speakers rated as advanced intermediate, 32 L1 Hindi speakers rated as advanced, 286 L1 Mandarin speakers rated as advanced intermediate, and 11 L1 Mandarin speakers rated as advanced.</div><div><br></div><div>Hierarchical Cluster Analysis (HCA) was conducted and presented four different clusters among all the L2 English speakers. The four different clusters are: (1) Low Mean Syllables per Run (MSR), low Speech Rate (SR), very high Pause Rate (PR), medium Measure of Textual Lexical Diversity (MTLD), and medium percentage of words on the Academic Word List (AWL); (2) Medium Mean Syllables per Run (MSR), medium Speech Rate (SR), high Pause Rate (PR), low Measure of Textual Lexical Diversity (MTLD), and low percentage of words on the Academic Word List (AWL); (3) High Mean Syllables per Run (MSR), high Speech Rate (SR), low Pause Rate (PR), medium Measure of Textual Lexical Diversity (MTLD), and medium percentage of words on the Academic Word List (AWL); (4) Medium Mean Syllables per Run (MSR), medium Speech Rate (SR), low Pause Rate (PR), very high Measure of Textual Lexical Diversity, and very high percentage level of words on the Academic Word List (AWL).</div><div>Chi-square results show that L2 English speakers’ cluster membership is strongly associated with both their L1 background and level of L2 oral English proficiency. While most of the advanced intermediate L1 Mandarin speakers are in Cluster 1 and Cluster 2, the majority of the advanced intermediate L1 Hindi speakers concentrate in Cluster 3. A large number of advanced L1 Mandarin speakers and L1 Hindi speakers are also located in Cluster 3.</div><div><br></div><div>Twelve raters were invited to evaluate speech samples representative of the four clusters in terms of accent difference and listener effort. Twelve speakers were selected from the four clusters, whose speech samples have values of the five linguistic features closest to the cluster mean.</div><div><br></div><div>Multi-facet Rasch Measurement (MFRM) results show that L1 Mandarin speakers generally received lower ratings in accent difference and listener effort. The connection among fluency, vocabulary, and accentedness/listener effort, however, functions differently for L1 Mandarin speakers and L1 Hindi speakers. For advanced intermediate L1 Mandarin speakers, those who speak slower and use more diverse vocabulary and more academic words were evaluated to be less accented, meanwhile costing less listener effort. However, advanced intermediate L1 Hindi speakers were rated as less accented and cost less listener effort when they demonstrate higher fluency measures and lower vocabulary measures.</div><div><br></div><div>Advanced L2 English speakers, in contrary, received reverse rating results. The advanced L1 Mandarin speaker, who speaks faster and uses less diverse vocabulary and fewer academic words, was evaluated to be less accented and cost less listener effort. However, the advanced L1 Hindi speaker, who speaks slower and uses more diverse vocabulary and more academic words, was rated as less accented and cost less listener effort.</div><div><br></div><div>This dissertation reemphasizes that holistic rating rubric does not deny the existence of multiple linguistic profiles. Raters are sensitive to different combinations of fluency and vocabulary features even if they have been asked to use a holistic scale. In addition, L2 English speakers may adopt individual strategies to accommodate while delivering, which calls for further pedagogical attention.<br></div><div><br></div>