The research reported here covers the following items: 1. The development of the Reflective Learning Framework (RLF). This framework was developed based on a grounded theoretical approach combined with an exploration of existing student reflections; then-current literature; and years of multiple iterations through real-world trials. This framework has now successfully been used by many educators in courses at McMaster and other institutions of higher education. 2. An investigation of students’ responses to the RLF. Following the development of the RLF, third-year students who had used the RLF to guide, assess, and evaluate their course reflections were surveyed and interviewed about their perceptions of using the RLF. Results of this study demonstrate that students see reflection as a tool to develop and use cognitive and metacognitive skills, and also as a tool to support knowledge retention and transfer. Furthermore, findings suggest how reflection, as studied, contributes to the acquisition of higher-order thinking skills required to address the complex challenges of improving sustainability. 3. Determining the effectiveness of the RLF. Finally, to determine the effectiveness of the RLF to support students in applying and demonstrating their higher-order thinking skills, a comparison of two sets of reflections was performed – one that used the RLF and one that did not. The comparison was based on tests of interrater reliability and a comparison of means was conducted. The results of this study demonstrate the reliability of the Framework to be used by multiple raters and that significantly different reflection outputs were obtained from students who used the RLF compared to those who did not. The results of these studies demonstrate the overall effectiveness of the RLF as a tool to guide, assess, and evaluate reflection through experiential education in university courses. / Thesis / Doctor of Philosophy (PhD) / The research reported here covers the following items: (1) the development of the Reflective Learning Framework (RLF), which was accomplished through a bottom-up approach, and which began with reading and re-reading over 100 student reflections and then reviewing the wealth of literature on the topics of reflection and experiential learning; (2) an investigation of students’ responses to the RLF, which was accomplished through an online survey and in-person interviews; and (3) determining the effectiveness of the RLF by comparing the grades of students who used the RLF to those who did not. The results of these studies demonstrate the overall effectiveness of the RLF as a tool to guide, assess, and evaluate reflection through experiential education in university courses.

This research clarified existing inconsistencies in self-assessment literature, and added to the body of knowledge for rating effectiveness of organizational assessments by defining relationships among rating effectiveness criteria (ratee, rater, rating scale, rating process) and measures (interrater reliability, halo error, leniency and severity, range restriction) based on extensive literature review. A research framework was developed from this review, and was employed in computing rating effectiveness measures at the individual (i.e., examiner or eight rating scale dimensions) and sector (e.g., Private Manufacturing Sector, Private Service Sector, Public Local Sector, Public State & Federal Sector) levels for a State Quality Award (SQA) using data from the 1998 applications. Interrater reliability (measured by intraclass correlations for each rating scale dimension) was low to moderate, and differed by dimension. Halo error (measured by the determinant of the dimension intercorrelation matrices for each examiner) was present for all examiners. Leniency and severity (measured by presence of statistically significant Rater main effect for each dimension) was present for 11 of 32 cases, and differed by dimension. Range restriction (measured by variance analysis for each dimension) was present for 22 of 32 cases, and differed by dimension. A post-hoc principle component analysis indicated poor internal reliability for the rating scale. To improve, the SQA should replace the existing rating scale and provide in-depth training on all elements of the rating process. The importance of the SQA using boxplots, histograms, and rating effectiveness measures to make fully informed decisions was discussed. / Master of Science

<p>Syfte: Undersöka den kriterierelaterade validiteten, mellanbedömarreliabiliteten och användarvänligheten för MEONF, ett screeninginstrument för bedömning av risk för undernäring. Metoder: Hundra patienter inkluderades och nutritionsstatus bedömdes med Mini Nutritional Assessment (MNA) som här anses vara golden standard samt screenades med Malnutrition Universal Screening Tool (MUST) och Minimal Eating Observation and Nutrition Form (MEONF). Resultat: Sjuksköterskorna ansåg MEONFs instruktioner och frågor lätta att förstå, svara på (100 %) samt att frågorna var angelägna (98 %). Instrumentet hade acceptabel sensitivitet (0.83) och specificitet (0.78) jämfört med MNA. Sensitiviteten var högre än den som fanns mellan MUST och MNA (0.57). När BMI ersattes med vadomkrets erhölls likvärdiga resultat (sensibilitet 0.83 specificitet 0.79) för MEONF i förhållande till MNA. Mellanbedömarreliabiliteten var hög (kappa 0.81). Sammanfattning: MEONF har bra validitet jämfört med MNA, med hög mellanbedömarreliabilitet. När BMI ersattes med vadomkrets bibehölls precisionen vilket är viktigt när vikt och längd är svåra att erhålla. Hög sensitivitet är angeläget vid screening för undernäring och i det avseendet fungerar MEONF bättre än MUST. En fundamental vinst med MEONF är att instrumentets kriterier upplevs som meningsfulla, för att vissa svårigheter kan avhjälpas med omvårdnadsåtgärder.</p><p>Syfte: Undersöka den kriterierelaterade validiteten, mellanbedömarreliabiliteten och användarvänligheten för MEONF, ett screeninginstrument för bedömning av risk för undernäring.</p><p>Metoder: Hundra patienter inkluderades och nutritionsstatus bedömdes med Mini Nutritional Assessment (MNA) som här anses vara <em>golden standard</em> samt screenades med Malnutrition Universal Screening Tool (MUST) och Minimal Eating Observation and Nutrition Form (MEONF).</p><p>Resultat: Sjuksköterskorna ansåg MEONFs instruktioner och frågor lätta att förstå, svara på (100 %) samt att frågorna var angelägna (98 %). Instrumentet hade acceptabel sensitivitet (0.83) och specificitet (0.78) jämfört med MNA. Sensitiviteten var högre än den som fanns mellan MUST och MNA (0.57). När BMI ersattes med vadomkrets erhölls likvärdiga resultat (sensibilitet 0.83 specificitet 0.79) för MEONF i förhållande till MNA. Mellanbedömarreliabiliteten var hög (kappa 0.81).</p><p>Sammanfattning: MEONF har bra validitet jämfört med MNA, med hög mellanbedömarreliabilitet. När BMI ersattes med vadomkrets bibehölls precisionen vilket är viktigt när vikt och längd är svåra att erhålla. Hög sensitivitet är angeläget vid screening för undernäring och i det avseendet fungerar MEONF bättre än MUST. En fundamental vinst med MEONF är att instrumentets kriterier upplevs som meningsfulla, för att vissa svårigheter kan avhjälpas med omvårdnadsåtgärder.</p> / <p>Rationale: Analyse the criterion related validity, interrater reliability and the usefulness of MEONF, an instrument for undernutrition risk screening. Methods: 100 patients were included and assessed with Mini Nutritional Assessment (MNA) regarded as golden standard and screened with Malnutrition Universal Screening Tool (MUST) as well as Minimal Eating Observation and Nutrition Form (MEONF). Results: Nurses regarded MEONF instructions and questions easy to understand, easy to answer (100%) and questions as being important (98%). It had acceptable sensitivity (0.83) and specificity (0.78) in relation to MNA. Sensitivity was higher than found between MUST and MNA (0.57). Replacing BMI with Calf Circumference (CC) gave equivalent results (sensibility 0.83, specificity 0.79) in relation to MNA. Interrater reliability was high (kappa 0.81). Conclusions: MEONF has good validity in relation to MNA and high interrater reliability. If replacing BMI with CC the precision is maintained, which is important in cases were weight or height is difficult to obtain. Having high sensitivity is a major concern for nutritional screening instruments and in that respect MEONF works better than MUST. One fundamental gain with MEONF is that the instruments criteria are experienced as clinically meaningful, in that some observed difficulties can be managed by nursing interventions.</p>

Syfte: Undersöka den kriterierelaterade validiteten, mellanbedömarreliabiliteten och användarvänligheten för MEONF, ett screeninginstrument för bedömning av risk för undernäring. Metoder: Hundra patienter inkluderades och nutritionsstatus bedömdes med Mini Nutritional Assessment (MNA) som här anses vara golden standard samt screenades med Malnutrition Universal Screening Tool (MUST) och Minimal Eating Observation and Nutrition Form (MEONF). Resultat: Sjuksköterskorna ansåg MEONFs instruktioner och frågor lätta att förstå, svara på (100 %) samt att frågorna var angelägna (98 %). Instrumentet hade acceptabel sensitivitet (0.83) och specificitet (0.78) jämfört med MNA. Sensitiviteten var högre än den som fanns mellan MUST och MNA (0.57). När BMI ersattes med vadomkrets erhölls likvärdiga resultat (sensibilitet 0.83 specificitet 0.79) för MEONF i förhållande till MNA. Mellanbedömarreliabiliteten var hög (kappa 0.81). Sammanfattning: MEONF har bra validitet jämfört med MNA, med hög mellanbedömarreliabilitet. När BMI ersattes med vadomkrets bibehölls precisionen vilket är viktigt när vikt och längd är svåra att erhålla. Hög sensitivitet är angeläget vid screening för undernäring och i det avseendet fungerar MEONF bättre än MUST. En fundamental vinst med MEONF är att instrumentets kriterier upplevs som meningsfulla, för att vissa svårigheter kan avhjälpas med omvårdnadsåtgärder. Syfte: Undersöka den kriterierelaterade validiteten, mellanbedömarreliabiliteten och användarvänligheten för MEONF, ett screeninginstrument för bedömning av risk för undernäring. Metoder: Hundra patienter inkluderades och nutritionsstatus bedömdes med Mini Nutritional Assessment (MNA) som här anses vara golden standard samt screenades med Malnutrition Universal Screening Tool (MUST) och Minimal Eating Observation and Nutrition Form (MEONF). Resultat: Sjuksköterskorna ansåg MEONFs instruktioner och frågor lätta att förstå, svara på (100 %) samt att frågorna var angelägna (98 %). Instrumentet hade acceptabel sensitivitet (0.83) och specificitet (0.78) jämfört med MNA. Sensitiviteten var högre än den som fanns mellan MUST och MNA (0.57). När BMI ersattes med vadomkrets erhölls likvärdiga resultat (sensibilitet 0.83 specificitet 0.79) för MEONF i förhållande till MNA. Mellanbedömarreliabiliteten var hög (kappa 0.81). Sammanfattning: MEONF har bra validitet jämfört med MNA, med hög mellanbedömarreliabilitet. När BMI ersattes med vadomkrets bibehölls precisionen vilket är viktigt när vikt och längd är svåra att erhålla. Hög sensitivitet är angeläget vid screening för undernäring och i det avseendet fungerar MEONF bättre än MUST. En fundamental vinst med MEONF är att instrumentets kriterier upplevs som meningsfulla, för att vissa svårigheter kan avhjälpas med omvårdnadsåtgärder. / Rationale: Analyse the criterion related validity, interrater reliability and the usefulness of MEONF, an instrument for undernutrition risk screening. Methods: 100 patients were included and assessed with Mini Nutritional Assessment (MNA) regarded as golden standard and screened with Malnutrition Universal Screening Tool (MUST) as well as Minimal Eating Observation and Nutrition Form (MEONF). Results: Nurses regarded MEONF instructions and questions easy to understand, easy to answer (100%) and questions as being important (98%). It had acceptable sensitivity (0.83) and specificity (0.78) in relation to MNA. Sensitivity was higher than found between MUST and MNA (0.57). Replacing BMI with Calf Circumference (CC) gave equivalent results (sensibility 0.83, specificity 0.79) in relation to MNA. Interrater reliability was high (kappa 0.81). Conclusions: MEONF has good validity in relation to MNA and high interrater reliability. If replacing BMI with CC the precision is maintained, which is important in cases were weight or height is difficult to obtain. Having high sensitivity is a major concern for nutritional screening instruments and in that respect MEONF works better than MUST. One fundamental gain with MEONF is that the instruments criteria are experienced as clinically meaningful, in that some observed difficulties can be managed by nursing interventions.

The purposes of this study were to: 1) determine absolute intrarater reliability of using the cervical range of motion device (CROM) for measuring cervical movements in older adults, and 2) determine the intrarater reliability and concurrent validity of the Candrive protocol, which uses a universal goniometer to measure rotation. Forty older adults (75.7 + 4.7 years of age) were tested in two sessions, one week apart, by two raters. Intrarater reliability scores were good for the CROM protocol (coefficient of variation (CV) values were 5.5% and 6.2 % for cervical rotation). The Candrive protocol values were higher (CV = 7.9 and 9.4%). Concordance analyses suggested that the Candrive protocol was less than good in terms of its validity, particularly when order effects were taken into consideration. In conclusion, the CROM protocol demonstrated good reliability for either group or individual analyses, whereas the Candrive protocol was less reliable and its validity marginal.

The purposes of this study were to: 1) determine absolute intrarater reliability of using the cervical range of motion device (CROM) for measuring cervical movements in older adults, and 2) determine the intrarater reliability and concurrent validity of the Candrive protocol, which uses a universal goniometer to measure rotation. Forty older adults (75.7 + 4.7 years of age) were tested in two sessions, one week apart, by two raters. Intrarater reliability scores were good for the CROM protocol (coefficient of variation (CV) values were 5.5% and 6.2 % for cervical rotation). The Candrive protocol values were higher (CV = 7.9 and 9.4%). Concordance analyses suggested that the Candrive protocol was less than good in terms of its validity, particularly when order effects were taken into consideration. In conclusion, the CROM protocol demonstrated good reliability for either group or individual analyses, whereas the Candrive protocol was less reliable and its validity marginal.

Bakgrund: Rehabiliteringen efter en rotatorkuffrupturoperation är oftast lång och följer olika faser med hänsyn till senans inläkning. Behandlingsresultatet är multifaktoriellt och det rekommenderas att rehabiliteringen individanpassas. För att följa och utvärdera rehabiliteringsprocessen behövs funktionella axeltester. I dagsläget saknas reliabla och validerade funktionella axeltester under rehabiliteringens tidigare faser. Syfte: Syftet med denna studie är att undersöka reliabiliteten hos två standardiserade armlyft, Aktivt armlyft i flexion och Aktivt armlyft i skapulaplanet, med statiskt kvarhåll i 90° under 30 sekunder, för vuxna individer som genomgått enrotatorkuffrupturoperation för cirka tre månader sedan. Metod: Datainsamlingen utfördes i tre steg, först utfördes en videoinspelning av de standardiserade armlyften av två fysioterapeuter. Därefter utfördes två bedömningstillfällen av två andra fysioterapeuter. Videoinspelningarna, armlyften och bedömningarna utfördes enligt standardiserade protokoll. Inter- och intrareliabilitetsbedömningarna baserades på videoinspelningarna där de två bedömarna graderade patienternas förmåga att utföra de två standardiserade armlyften utifrån utfallsmåtten ”Klarar” eller ”Klarar ej”. Överensstämmelsen analyserades i IBM SPSS med Cohens Kappa. Resultat: Åtta manliga studiedeltagare i åldrarna 49–71 år (medelålder 59,4 år) deltog. Videoinspelningarna utfördes i genomsnitt 15,25 (±3,9) veckor efter operation. Den andra bedömningen genomfördes i genomsnitt 9 (±3,1) dagar efterförsta bedömningen. Resultatet visade att interbedömarreliabiliteten var ”God”(k=0,71) till ”Utmärkt” (k=1,00) och intrabedömareliabiliteten var ”Utmärkt”(k=1,00). Slutsats: Denna studie fann att inter- och intrabedömarreliabilitet var ”God” till”Utmärkt” gällande de två standardiserade armlyften. Resultatet bör tolkas med försiktighet på grund av det begränsade deltagarantalet och ingen säker slutsats kandras. Ytterligare forskning behövs för att fastställa klinisk betydelse. / Background: Rehabilitation after rotator cuff tear surgery is often long and follows different phases with consideration for tendon healing. The treatment result is multifactorial, and the rehabilitation is recommended to be individualized.vFunctional shoulder tests are of value and evaluate the rehabilitation process. Currently, reliable and validated functional shoulder tests are lacking during the earlier phases of rehabilitation. Purpose: The purpose of this study is to investigate the reliability of two standardized arm lifts, Active shoulder flexion and Active shoulder scaption, with astatic hold at 90° for 30 seconds, three months post-surgery. Method: Data collection was conducted in three steps. Starting with videorecordings of the standardized arm lifts by two physiotherapists. Followed by two assessment sessions with approximately one week apart, by two other physiotherapists. The video recordings, arm lifts, and assessments were performed according to standardized protocols. Inter- and intrarater reliability assessments were based on the video recordings. The examiners graded the patient’s ability to perform the two functional shoulder tests based on the outcomes "Approved" or "Non-approved". The agreement was calculated with IBM SPSS using Cohen's Kappa. Results: Eight men were included, 49-71 years (mean age 59,4 years). The videorecordings and the first evaluation were performed in mean 15,3 (±3,9) weeks postsurgery. The second evaluation was performed in mean 9,4 (±3,1) days after the first evaluation. The results showed that the interrater-reliability was “Good” (k=0,71) to“Excellent” (k=1,00), and the intrarater-reliability was “Excellent” (k=1,00). Conclusion: This study found that the inter- and intrarater reliability for the two standardized arm lifts were good to excellent. The limited number of participants requires caution with the interpretation of the results and no definitive conclusion can be drawn. Further research is needed to establish any clinical significance.

Background: The Swedish driving licence test is a criterion-referenced test resulting in a pass or fail. It currently consists of two parts - a theory test with 65 multiple-choice items and a practical driving test where at least 25 minutes are spent driving in traffic. It is a high-stakes test in the sense that the results are used to determine whether the test-taker should be allowed to drive a car without supervision. As the only other requirements for obtaining a licence is a few hours of hazard education (and a short introduction if you intend to drive with a lay instructor) it is important that the test result, in terms of pass or fail, is reliable and valid. If this is not the case it could have detrimental effects on traffic safety. Examining all relevant aspects is beyond the scope of this licentiate thesis so I have focused on reliability. Methods Reliability for both the theoretical and practical test results was examined. As these are very different types of tests the types of reliability examined also differed. In order to examine inter-rater reliability of the driving test 83 examiners were accompanied by one of five selected supervising examiners for a day of tests. All in all 535 tests were conducted with two examiners assessing the same performance. At the end of the day the examiners compared notes and tried to determine the reason for any inconsistencies. Both examiners and students also filled in questionnaires with questions about background and preparation. As for studying decision consistency and decision accuracy of the theory test, three test versions (a total of around 12,000 tests) were examined with the help of methods devised by Subkoviak (Subkoviak, 1976, 1988) and Hanson &amp; Brennan (Brennan, 2004; Hanson &amp; Brennan, 1990). Results The results from two research studies concerning reliability were presented. Study I focused on inter-rater reliability in the driving test and in 93 per cent of cases the examiners made the same assessment. For the tests where their opinions differed there was no correlation to any of the background variables or other variables examined except for three, which had logical explanations and did not constitute a problem. Although there were cases where the differences were due to different stances on matters of interpretation the most common suggested cause was the placement in the car (back seat vs. front seat). Although the supervising examiners gave both praise and criticism as to how the test was carried out the study does not answer the question whether the tests were equal in terms of composition and difficulty. In Study II the focus was on decision consistency and decision accuracy in the theory test. Three versions of the theory tests were examined and, on the whole, found to be fairly similar in terms of item difficulty and score distribution, but the mean was so close to the cut-score (i.e. the score required to pass) that the pass rate differed somewhat between versions. Agreement coefficients were around .80 for all test versions (between .79 and .82 depending on method). Classification accuracy indicated an .87 probability of a correct classification. Conclusion It is important to examine the reliability and validity of the driving licence test since a misclassification can have serious consequences in terms of traffic safety. In the studies included here the rate of agreement between examiners is deemed as satisfactory. It would be preferable if the classification consistency and classification accuracy, as estimated by the methods used, were higher for the theory test, given its importance. While reliability in terms of agreement between raters/examiners or consistency and accuracy of classification are routinely examined in other contexts, such as large-scale educational testing, this is not often done for the driving licence tests. At the same time, the methods used here can be transferred to contexts where such properties are generally not examined. Collecting information about test-takers and examiners, like in Study I, can provide evidence concerning possible bias. Examining to what extent decisions are consistent is one important aspect of collecting evidence that shows that test results can be used to draw conclusions about driver competence. Still, regardless of outcome, validation is a process that never ends. There is always reason to examine various aspects and make further improvements. There are also many other relevant aspects to examine. A prerequisite for the validity of the score interpretation of a criterion-referenced test like this one is that the cut-score is appropriate and the content relevant. This should therefore be the subject of further research as the validation process continues.

The purpose of this study was to investigate if early childhood educators could reliably score items using a new scoring system for the Assessment, Evaluation, and Programming System for Infants and Children (AEPS; Bricker, 2002). The participants were university students completing their certification in Interdisciplinary Early Childhood Education (IECE) at the University of Kentucky (UK). The six participants completed training on implementing the AEPS and administered the AEPS to measure child outcomes. The results of this study validated the new scoring system for the AEPS by illustrating that the participants could reliably score a curriculum based assessment.

No description available.