Abstract The objective was to assess Professional, Health Advocate and Scholar CanMEDS competencies in anesthesia residents using a Generic Integrated Objective Structured Assessment Tool (GIOSAT) during simulated scenarios. Methods Twenty one anesthesia residents participated in managing two scenarios: “do not resuscitate” and “morphine overdose”. Four trained blinded raters analyzed video recordings using the GIOSAT. Internal structure was examined using generalizability analysis. Results Results of the g-study focused on PHAS and ME components alone, participants accounted for 20% of the scores variance (G-coefficient 0.66). A D-study indicated two raters and eleven or more scenarios would be required for a G-coefficient > 0.80. Conclusion This study demonstrates construct validity for assessing PHAS competencies for low stakes assessment. Results address the gap of assessment performance for PHAS competencies, describe methodology and produce recommendations for summative assessments using simulation.

The study of interrater agreement and itnerrater reliability attract extensive attention, due to the fact that the judgments from multiple raters are subjective and may vary individually. To evaluate interrater agreement and interrater reliability, five different methods or indices are proposed: percentage of agreement, kappa coefficient, the correlation coefficient, intraclass correlation coefficient (ICC), and generalizability (G) theory. In this study, we introduce and discuss the advantages and disadvantages of these methods to evaluate interrater agreement and reliability. Then we review and explore the rank across these five indices by use of frequency in practice in the past five years. Finally, we illustrate how to use these five methods under different circumstances and provide SPSS and SAS code to analyze interrater agreement and reliability. We apply the methods above to analyze the data from Parent-Child Interaction System of global ratings (PARCHISY), and conclude as follows: (1) ICC is the most often used method to evaluate interrater reliability in recent five years, while generalizability theory is the least often used method. The G coefficients provide similar interrater reliability with weighted kappa and ICC on most items, based on the criteria. (2) When the reliability is high itself, different methods provide consistent indication on interrater reliability based on different criteria. If the reliability is not consistent among different methods, both ICC and G coefficient will provide better interrater reliability based on the criteria, and they also provide consistent results.

Many sports biomechanics research studies follow a traditional task analysis concept that there is only one best possible movement pattern and thus focus on the examination of kinematics and kinetics of movement without considering the influence of constraints that are imposed on it. This study developed an interdisciplinary approach by utilizing the principles of ecological task analysis and movement coordination from areas of motor leaning and biomechanics to examine the skill of placement hitting in slo-pitch softball. The choice of evaluating this slo-pitch batting skill to assess movement patterns is pragmatic because of its popularity of the sport and uniqueness of the batting movement. Therefore, the purpose of this study was to examine the influence of two task constraints (stride technique and designated field location) and an environmental constraint (pitched ball location) on the participants batting performances, kinematics, and movement patterns. A three-way ANOVA of 2 fields (same and opposite) x 2 locations of pitch (inside and outside) x 3 strides (open, parallel and closed) repeated measure study was conducted in this study. The results showed that participants were more successful in placing the ball to the same field instead of the opposite field. The pitched ball location and stride techniques did not have a consistent impact on the results across the different hitting conditions. To achieve these batting performance results, participants demonstrated different joint movements and different coordination patterns. Hence, this study supports the rationale of ecological task analysis but not traditional task analysis. Further, to understand the generalizability of the findings, a Euclidean distance analysis was conducted to evaluate the degree of dissimilarity between the individual and group mean results. The results indicated that participants generally showed a low degree of dissimilarity, so they were quite homogeneous as a group. Hence, the results from this study not only enable us to evaluate a human movement skill under the influence of different constraints but educators may apply the findings to other players. A similar interdisciplinary approach is warranted for future research studies in order to better understand the mechanics of human motion.

Abstract The objective was to assess Professional, Health Advocate and Scholar CanMEDS competencies in anesthesia residents using a Generic Integrated Objective Structured Assessment Tool (GIOSAT) during simulated scenarios. Methods Twenty one anesthesia residents participated in managing two scenarios: “do not resuscitate” and “morphine overdose”. Four trained blinded raters analyzed video recordings using the GIOSAT. Internal structure was examined using generalizability analysis. Results Results of the g-study focused on PHAS and ME components alone, participants accounted for 20% of the scores variance (G-coefficient 0.66). A D-study indicated two raters and eleven or more scenarios would be required for a G-coefficient > 0.80. Conclusion This study demonstrates construct validity for assessing PHAS competencies for low stakes assessment. Results address the gap of assessment performance for PHAS competencies, describe methodology and produce recommendations for summative assessments using simulation.

No description available.

Humans are constantly faced with learning motor tasks throughout their lifespan (e.g., children learning how to throw a ball overhand, elite athletes learning how to become more even more efficient at their sports performance, and an older adult relearning how to walk post-stroke recovery). With such variety in the types of motor tasks that humans try to learn across the lifespan, little is known about the impact of a learner’s previous motor skill experience. Thus, the purpose of this thesis was to investigate when motor learning generalizability or specificity are more likely to occur, respectively. An in-depth background of motor learning generalizability and specificity was provided in chapter one. The scope of the motor learning literature including generalizability and/or specificity was investigated in chapter two. At the end of chapter two, certain limitations of the motor learning literature are addressed and framed into a useable checklist for future motor learning experiments. Chapter three serves as a bridging chapter to connect the scoping review and checklist in chapter two, to the framework implemented in chapter four. In chapter four, the checklist was employed to assess its usefulness in future motor learning experiments. Collectively, this thesis provides organization to the previous motor learning generalizability and specificity literature, as well as recommendations for future motor learning researchers based on a tested framework protocol. / Dissertation / Doctor of Philosophy (PhD) / Our previous movement experiences can impact our capability to learn new motor tasks. These previous movement experiences can be either beneficial or detrimental (or have no effect) on our learning of that task depending on many different things with no real definitive answers to why the outcomes differ and when. The purpose of this thesis is to review how prior motor skill practice may be beneficial to future motor skill learning (generalizability), detrimental to learning, or no effect (specificity) and to organize these findings into a new ‘types of transfer’ taxonomy, create a framework to help guide future motor learning research and conduct an experiment that follows this framework. By considering and organizing this large motor learning literature into a review, creating this taxonomy and outlining an empirical investigative framework, this thesis will help us to better understand motor learning history and provide a pathway forward for future researchers.

No description available.

Research in interactive information retrieval (IR) usually relies on lab user studies or online ones. A key concern of these studies is the generalizability and reproducibility of the results, especially when the studies involved only a limited number of participants. The interactive IR community, however, does not have a commonly agreed guideline regarding how many participants should recruit. We study this fundamental research protocol issue by examining the generalizability and reproducibility of results with respect to a different number of participants using simulation-based approaches. Specifically, we collect a relatively large number of participants' observations for a representative interactive IR experiment setting from online user studies using crowdsourcing. We sample smaller numbers of participants' results from the collected observations to simulate the results of online user studies with a smaller scale. We empirically analyze the patterns of generalizability and reproducibility regarding different dependent variables and draw conclusions related to the optimal number of participants. Our study contributes to interactive information retrieval research by 1) establishing a methodology for evaluating the generalizability and reproducibility of results, and 2) providing guidelines regarding the optimal number of participants for search engine user studies. / Master of Science / In the domain of Information Retrieval, researchers or scientists usually require human participants to interact, test and evaluate a novel system, which is usually called user studies. However, researchers usually perform these studies with small sample size, some of them recruited fewer than 20 participants, which casts doubt on the generalizability and reproducibility of these studies. Generalizability means how reliable the results of relatively small sample size in an experimental setting can be generalized to the outcomes of a larger population. Reproducibility means whether the results from two groups with the same amount of sample size are consistent with each other. In order to examine the generalizability and reproducibility of online user studies in interactive information retrieval systems, we conducted an online user study with large sample size. We reproduced a well-recognized lab user study from Kelly et al. (2015) in an online environment. We established a simulation-based methodology for evaluating the generalizability and reproducibility of the results and then provided guidelines regarding the optimal number of participants for search engine user studies.

Work-related musculoskeletal disorders (WMSDs) cost US industries billions annually and reduce quality of life for those afflicted. Passive exoskeletons (EXOs) have emerged as a potential intervention to reduce worker exposures to WMSD risk factors. As EXO adoption is rising, EXO manufacturers are designing and producing new EXOs in accordance with growing demand. However, there are no standardized EXO evaluation protocols and EXO use recommendations, due in part to insufficient information on the reliability of EXO evaluation measures. The purpose of this thesis was to quantify the reliability of common EXO evaluation measures, using both traditional approaches a more advanced statistical approach (i.e., Generalizability Theory), while also identifying potential effects of EXO type, work task, and individual differences. This work used data from a recently completed EXO evaluation study, conducted in Virginia Tech's Occupational Ergonomics and Biomechanics Lab. Forty-two total participants completed simulated occupational tasks, in two separate experimental sessions on different days, while using an arm-support EXO (ASE) and a back-support EXO (BSE). Several outcome measures reached excellent within-session reliability within four trials for many tasks considered. Between-session reliability levels were lower than within-session levels, with outcome measures reaching moderate-to-good reliability for most tasks. Interindividual differences accounted for the largest proportion of variance for measurement reliability, followed by the experimental session. For all tasks, outcome measures reached excellent dependability levels, with many achieving excellent levels within five total trials. Inconsistencies observed in between-session reliability levels and dependability levels suggest that additional training and EXO familiarity may affect measurement reliability of outcome measures differently for some tasks, unique to each EXO type. These discrepancies emphasize the importance for additional research into this topic. Overall, the current findings indicate that many of the commonly used EXO evaluation measures are reliable and dependable within five trials and one experimental session, providing a potential foundation for standardized EXO assessment protocols. / Master of Science / Work-related musculoskeletal disorders (WMSDs) are a substantial economic burden and impair the quality of life for affected workers. Passive exoskeletons (EXOs), which use springs or elastic material to distribute the load placed on workers during manual labor, are a possible solution to reduce worker exposure to WMSD risk factors. EXO adoption is rising, but there are no standardized procedures to test the effectiveness of EXOs or standardized recommendations for EXO use. The purpose of this thesis was to determine the reliability of EXO evaluation measures commonly used in prior research, using both traditional reliability calculation methods alongside a more advanced method (i.e., Generalizability Theory). Data from a recently completed study were used, which were collected from 42 participants in two separate experimental sessions on two different days. Participants completed tasks intended to simulate manual work, using either an arm-support exoskeleton – which supported their upper arms during relevant tasks, or a back-support exoskeleton – which supported their lower back during relevant tasks. Many of the tasks and outcome measures reached excellent reliability within four repetitions in a single day. When examining reliability of evaluations across days, we found reliability levels were lower than levels obtained from a single day. All tasks and outcome measures reached excellent dependability levels, with many requiring only five trials to reach excellent levels. Reliability increased with the number of trials in an EXO evaluation experiment. Moreover, our results revealed that the EXO type being used and the biological sex of a participant both influence reliability, but individual participant differences had the greatest effect on measurement reliability. This research reveals possible experimental conditions required for reliable, efficient, and cost-effective EXO research, facilitating the development of a standardized EXO evaluation protocol.

Many social processes influence the amount, quality, and availability of support from an individual's social network. Trait influences are characteristics of the individual that generalize across relationships and affect how much support is received and perceived on average from other people. Social influences comprise characteristics of the individual's social network. They are relationship specific and account for the variability in supportiveness among an individual's providers. Recent studies have taken a multilevel approach to studying social support in order to partition the variance in sets of relationship-specific support measures into trait and social components. These studies have also used multivariate generalizability (G) theory to examine the correlations between social support and other constructs, such as negative mood, at the trait and social level. These multilevel studies have begun to clarify the relative contributions of trait and social influences on social support, but much is yet to be learned about the nature and measurement of social support's trait and social components. One set of aims within this project was to identify characteristics of support recipients and characteristics of support providers that were related to the reception and perception of social support. Another set of aims focused on validating the measurement strategies used by G theory researchers and understanding how the trait and social components of support and mood derived from relationship-specific measures relate to traditional measures of these constructs. My final set of aims involved the application of multilevel analyses of social support and negative mood to three existing theories in the social support literature--the buffering hypothesis, the matching hypothesis, and the platinum rule. The participants in this study comprised two samples--one group of 755 undergraduate psychology students, and one group of 430 community members from across the United States. Participants completed measures of their personality traits, recent depressive symptoms, recent experiences of life adversity and perceived control over life adversity. They also reported on three close relationships including support from those relationships, satisfaction with those relationships, and mood experienced when interacting with those three people. Several multilevel analyses were used in the study. Univariate G theory analyses were used to quantify the relative variance in support, mood, and relationship satisfaction attributable to trait and social influences. Multivariate G theory analyses were used to estimate the links between these variables at the trait and social levels of analysis. Mixed effects models were used to identify trait and relationship-specific constructs that that might partly constitute the trait and social influences on social support. Multilevel Structural Equation Modeling (SEM) was used to evaluate the validity of several constructs employed in previous multilevel studies on social support. Finally, mixed effects and multivariate G theory analyses were used to test the buffering hypothesis, the matching hypothesis, and the platinum rule. Consistent with previous multilevel studies of social support, recipients who received more support, on average, from their social networks also reported more negative mood when interacting with their providers. After taking those average tendencies into account, the amount of support received from an individual support provider was not associated with negative mood experienced when with that provider. The investigation of the trait influences on social support showed that recipients who were younger, more extraverted, and more open to new experiences tended to receive more social support. Women tended to receive more support than men. With respect to social influences, romantic partners tended to provide the most support whereas friends and siblings provided significantly less support on average. Women tended to provide more support than men. The validity assessment showed that the social component of support availability was only modestly distinct from the social component of generic relationship satisfaction. The trait component of support availability showed good discriminant validity from relationship satisfaction and good convergent validity with global support availability. The trait component of relationship-specific mood showed moderate convergent validity with general mood. The buffering and matching hypotheses were not supported by my findings. The platinum rule was supported at the trait level in that recipients who reported greater support adequacy, on average, tended to report more positive mood and less negative mood. The platinum rule was also supported at the social level in that recipients tended to report experiencing the most positive mood and least negative mood when interacting with individual providers who tended to supply the most adequate support.