Data-driven decision making is an integral part of higher education and it needs to be rooted in strong methodological and statistical practices. Key practices include the use and interpretation of effect sizes as well as a correct understanding of null hypothesis significance testing (NHST). Therefore, effect size reporting and interpreting practices in higher education journal articles represent an important area of inquiry. This study examined effect size reporting and interpretation practices of published quantitative studies in three core higher education journals: Journal of Higher Education, Review of Higher Education, and Research in Higher Education. The review covered a three-year publication period between 2013 and 2015. Over the three-year span, a total of 249 articles were published by the three journals. The number of articles published across the three years did not vary appreciably. The majority of studies employed quantitative methods (71.1%), about a quarter of them used qualitative methods (25.7%), and the remaining 3.2% used mixed methods. Seventy-three studies were removed from further analysis because they did not feature any quantitative analyses. The remaining 176 quantitative articles represented the sample pool. Overall, 52.8% of the 176 studies in the final analysis reported effect size measures as part of their major findings. Of the 93 articles reporting effect sizes, 91.4% of them interpreted effect sizes for their major findings. The majority of studies that interpreted effect sizes also provided a minimal level of interpretation (60.2% of the 91.4%). Additionally, 26.9% of articles provided average effect size interpretation, and the remaining 4.3% of studies provided strong interpretation and discussed their findings in light of previous studies in their field.

To help address the knowledge gap between science and practice this study explores the possible positive benefits of using a more pedagogical effect size estimate when presenting statistical relationships. Traditional presentation has shown limitations with major downsides being that scientific findings are misinterpreted or misunderstood even by professionals. This study explores the possible effects of the non-traditional effect size estimate Common Language Effect Size (CLES) on different training outcomes for HR professionals. This study also explores the possible effect of cognitive system preference on training outcomes. Results show no overall effect of CLES on either training outcomes or cognitive system. A positive effect of CLES on training outcome is found at the subfactor level showing a significant effect. The results can be interpreted that non-traditional effect size estimates have a limited effect on training outcomes. This small but valuable piece to bridge the gap of knowledge is discussed.

Master of Science / Department of Statistics / Paul Nelson / Effect size is a concept that was developed to bridge the gap between practical and statistical significance. In the context of completely randomized one way designs, the setting considered here, inference for effect size has only been developed under normality. This report is a simulation study investigating the robustness of nominal 0.95 confidence intervals for effect size with respect to departures from normality in terms of their coverage rates and lengths. In addition to the normal distribution, data are generated from four non-normal distributions: logistic, double exponential, extreme value, and uniform. The report discovers that the coverage rates of the logistic, double exponential, and extreme value distributions drop as effect size increases, while, as expected, the coverage rate of the normal distribution remains very steady at 0.95. In an interesting turn of events, the uniform distribution produced higher than 0.95 coverage rates, which increased with effect size. Overall, in the scope of the settings considered, normal theory confidence intervals for effect size are robust for small effect size and not robust for large effect size. Since the magnitude of effect size is typically not known, researchers are advised to investigate the assumption of normality before constructing normal theory confidence intervals for effect size.

abstract: To make meaningful comparisons on a construct of interest across groups or over time, measurement invariance needs to exist for at least a subset of the observed variables that define the construct. Often, chi-square difference tests are used to test for measurement invariance. However, these statistics are affected by sample size such that larger sample sizes are associated with a greater prevalence of significant tests. Thus, using other measures of non-invariance to aid in the decision process would be beneficial. For this dissertation project, I proposed four new effect size measures of measurement non-invariance and analyzed a Monte Carlo simulation study to evaluate their properties and behavior in addition to the properties and behavior of an already existing effect size measure of non-invariance. The effect size measures were evaluated based on bias, variability, and consistency. Additionally, the factors that affected the value of the effect size measures were analyzed. All studied effect sizes were consistent, but three were biased under certain conditions. Further work is needed to establish benchmarks for the unbiased effect sizes. / Dissertation/Thesis / Doctoral Dissertation Psychology 2019

Patient reported outcomes (PROs) are becoming increasingly important in cancer studies, particularly with the emphasis on patient centered outcome research. However, multiple PROs, using different scales, with different directions of favorability are often used within a trial, making interpretation difficult. To enhance interpretability, we propose the use of a standardized effect size graph, which shows all PROs from a study on the same figure, on the same scale. Plotting standardized effects with their 95% confidence intervals (CIs) on a single graph clearly showing the null value conveys a comprehensive picture of trial results. We demonstrate how to create such a graph using data from a randomized controlled trial that measured 12 PROs at two time points. The 24 effect sizes and CIs are shown on one graph and clearly indicate that the intervention is effective and sustained.

Research pertaining to the distortion of the squared canonical correlation coefficient has traditionally been limited to the effects of sampling error and associated correction formulas. The purpose of this study was to compare the degree of attenuation of the squared canonical correlation coefficient under varying conditions of score reliability. Monte Carlo simulation methodology was used to fulfill the purpose of this study. Initially, data populations with various manipulated conditions were generated (N = 100,000). Subsequently, 500 random samples were drawn with replacement from each population, and data was subjected to canonical correlation analyses. The canonical correlation results were then analyzed using descriptive statistics and an ANOVA design to determine under which condition(s) the squared canonical correlation coefficient was most attenuated when compared to population Rc2 values. This information was analyzed and used to determine what effect, if any, the different conditions considered in this study had on Rc2. The results from this Monte Carlo investigation clearly illustrated the importance of score reliability when interpreting study results. As evidenced by the outcomes presented, the more measurement error (lower reliability) present in the variables included in an analysis, the more attenuation experienced by the effect size(s) produced in the analysis, in this case Rc2. These results also demonstrated the role between and within set correlation, variable set size, and sample size played in the attenuation levels of the squared canonical correlation coefficient.

The self-correcting nature of psychological and educational science has been seriously questioned. Recent special issues of Perspectives on Psychological Science and Psychology of Aesthetics, Creativity, and the Arts have roundly condemned current organizational models of research and dissemination and have criticized the perverse incentive structure that tempts researchers into generating and publishing false positive findings. At the same time, replications are rarely attempted, allowing untruths to persist in the literature unchallenged. In this article, the editors of the Journal of Advanced Academics consider this situation and announce new policies for quantitative submissions. They are (a) an explicit call for replication studies; (b) new instructions directing reviewers to base their evaluation of a study’s merit on the quality of the research design, execution, and written description, rather than on the statistical significance of its results; and (c) an invitation to omit statistical hypothesis tests in favor of reporting effect sizes and their confidence limits.

It was hypothesized that effect size in therapy outcome research would correlate positively with congruence effects. Congruence was defined as the degree to which what had been practiced in treatment was scored as improvement when outcome was measured. Additionally, it was hypothesized that correcting effect sizes for estimated nongeneralizable change attributable to congruence (i.e., representativeness reduction) would significantly reduce the average magnitude of effect.

No description available.

No description available.