The overall effectiveness of computer-assisted instruction (CAI) for higher order learning in technical education and training was determined through the meta-analysis approach. Studies that had investigated the effectiveness of CAI as compared to traditional instruction were selected from major databases in the civilian and military sectors. The selection criteria were: (a) instruction was in the area of technical education and training, (b) a comparison was made between a group of students that received computer-assisted instruction with another group that was taught in the traditional manner, (c) student learning in both groups was measured in some form, and (d) quantitative results on criterion measures were provided. The common comparison metric chosen to indicate the effect size was the standardized mean difference. Additionally, a determination was made of the difference in CAI effectiveness between studies categorized into: (a) CAI type -- intelligent CAI and ordinary CAI; (b) nature of CAI treatment -- replacement and supplemental; (c) subject assignment -- random groups, intact groups, and assignments other than the preceding two groups; (d) educational level -- secondary / postsecondary, university, and adult military training; and (e) setting -- civilian and military. The overall effect size of CAI was found to be 0.35, implying that the average student in the traditional class would have improved from the 50th percentile to the 64th percentile if the student had been provided with CAI. Intelligent CAI was found to be significantly more effective than ordinary CAI. / Ph. D.

No description available.

The shortcomings of the proportion above cut (PAC) statistic used so prominently in the educational landscape renders it a very problematic measure for making correct inferences with student test data. The limitations of PAC-based statistics are more pronounced with cross-test comparisons due to their dependency on cut-score locations. A better alternative is using mean-based statistics that can translate to parametric effect-size measures. However, these statistics as well can be problematic. When Gaussian assumptions are not met, reasonable transformations of a score scale produce non-monotonic outcomes. The present study develops a distribution-wide approach to summarize trend, gap, and gap trend (TGGT) measures. This approach counters the limitations of PAC-based measures and mean-based statistics in addition to addressing TGGT-related statistics in a manner more closely tied to both the data and questions regarding student achievement. This distribution-wide approach encompasses visual graphics such as percentile trend displays and probability-probability plots fashioned after Receiver Operating Characteristic (ROC) curve methodology. The latter is framed as the P-P plot framework that was proposed by Ho (2008) as a way to examine trends and gaps with more consideration given to questions of scale and policy decisions. The extension in this study involves three main components: (1) incorporating Bayesian inference, (2) using a multivariate structure for longitudinal data, and (3) accounting for measurement error at the individual level. The analysis is based on mathematical assessment data comprising Grade 3 to Grade 7 from a large Midwestern school district. Findings suggest that PP-based effect sizes provide a useful framework to measure aggregate test score change and achievement gaps. The distribution-wide perspective adds insight by examining both visually and numerically how trends and gaps are affected throughout the score distribution. Two notable findings using the PP-based effect sizes were (1) achievement gaps were very similar between the Focal and Audit test, and (2) trend measures were significantly larger for the Audit test. Additionally, measurement error corrections using the multivariate Bayesian CTT approach had effect sizes disattenuated from those based on observed scores. Also, the ordinal-based effect size statistics were generally larger than their parametric-based counterparts, and this disattenuation was practically equivalent to that seen by accounting for measurement error. Finally, the rank-based estimator of P(X>Y) via estimated true scores had smaller standard errors than for its parametric-based counterpart.

Brief experimental analysis (BEA) is a well-documented analysis strategy that rapidly manipulates instructional variables to identify the most effective intervention to support a student’s academic needs. However, consensus on how BEA data should be evaluated is not evident in published BEA articles. This study investigated the agreement between evaluation methods (i.e., visual analysis, no assumptions effect size, percentage of nonoverlapping data, nonoverlap of all pairs) used in BEA. Overall, the measures of effect size resulted in a higher percentage of positive agreement with other measures of effect size, in comparison to visual analysis paired with effect size measures. Use of effect size measures also resulted in less equivalency between intervention outcomes within a BEA. These data suggest that using a measure of effect size can be a beneficial component to visual analysis; however, each measure of effect size has its own strengths and limitations and should be used cautiously when interpreting results of a BEA.

ABSTRACT A MONTE CARLO STUDY INVESTIGATING MISSING DATA, DIFFERENTIAL ITEM FUNCTIONING, AND EFFECT SIZE by Phyllis Garrett The use of polytomous items in assessments has increased over the years, and as a result, the validity of these assessments has been a concern. Differential item functioning (DIF) and missing data are two factors that may adversely affect assessment validity. Both factors have been studied separately, but DIF and missing data are likely to occur simultaneously in real assessment situations. This study investigated the Type I error and power of several DIF detection methods and methods of handling missing data for polytomous items generated under the partial credit model. The Type I error and power of the Mantel and ordinal logistic regression were compared using within-person mean substitution and multiple imputation when data were missing completely at random. In addition to assessing the Type I error and power of DIF detection methods and methods of handling missing data, this study also assessed the impact of missing data on the effect size measure associated with the Mantel, the standardized mean difference effect size measure, and ordinal logistic regression, the R-squared effect size measure. Results indicated that the performance of the Mantel and ordinal logistic regression depended on the percent of missing data in the data set, the magnitude of DIF, and the sample size ratio. The Type I error for both DIF detection methods varied based on the missing data method used to impute the missing data. Power to detect DIF increased as DIF magnitude increased, but there was a relative decrease in power as the percent of missing data increased. Additional findings indicated that the percent of missing data, DIF magnitude, and sample size ratio also influenced the effect size measures associated with the Mantel and ordinal logistic regression. The effect size values for both DIF detection methods generally increased as DIF magnitude increased, but as the percent of missing data increased, the effect size values decreased.

Standardized testing has been part of the American educational system for decades. Controversy from the beginning has plagued standardized testing, is plaguing testing today, and will continue to be controversial. Given the current federal educational policies supporting increased standardized testing, psychometricians, educators and policy makers must seek ways to ensure that tests are not biased towards one group over another. In measurement theory, if a test item behaves differently for two different groups of examinees, this test item is considered a differential functioning test item (DIF). Differential item functioning, often conceptualized in the context of item response theory (IRT) is a term used to describe test items that may favor one group over another after matched on ability. It is important to determine whether an item is functioning significantly different for one group over another regardless as to why. Hypothesis testing is used to determine statistical significant DIF items; an effect size measure quantifies a statistical significant difference. This study investigated the addition of reporting an effect size measure for differential item functioning of items and tests’ (DFIT) noncompensatory differential item functioning (NCDIF), and reporting empirically observed power. The Mantel-Haenszel (MH) parameter served as the benchmark for developing NCDIF’s effect size measure, for reporting moderate and large differential item functioning in test items. In addition, by modifying NCDIF’s unique method for determining statistical significance, NCDIF will be the first DIF statistic of test items where in addition to reporting an effect size measure, empirical power can also be reported. Furthermore, this study added substantially to the body of literature on effect size by also investigating the behavior of two other DIF measures, Simultaneous Item Bias Test (SIBTEST) and area measure. Finally, this study makes a significant contribution to the body of literature by verifying in a large-scale simulation study, the accuracy of software developed by Roussos, Schnipke, and Pashley (1999) to calculate the true MH parameter. The accuracy of this software had not been previously verified.

<p>Diss. (sammanfattning) Umeå : Umeå universitet, 2002, Härtill 5 uppsatser.</p> / digitalisering@umu

In this thesis,we study how a meta-analysis of genetic association studies is influenced by the degree of genotype imputation uncertainty in the studies combined and the size of meta-analysis. We consider the fixed effect meta-analysis model to evaluate the accuracy and efficiency of imputation-based meta-analysis results under different levels of imputation accuracy. We also examine the impact of genotype imputation on the between-study heterogeneity and type 1 error in the random effects meta-analysis model. Simulation results reaffirm that meta-analysis boosts the power of detecting genetic associations compared to individual study results. However, the power deteriorates with increasing uncertainty in imputed genotypes. Genotype imputation affects a random effects meta-analysis in a non-obvious way as estimation of between-study heterogeneity and interpretation of association results depend heavily on the number of studies combined. We propose an adjusted fixed effect meta-analysis approach for adding imputation-based studies to a meta-analysis of existing typed studies in a controlled way to improve precision and reliability. The proposed method should help in designing an effective meta-analysis study.

Over the past months, newspapers and news bulletins are riddled by service delivery protests – to the extent that the President of the country acknowledges these protests. Many reasons were offered for the protests, but the main reason is cited as dissatisfaction with the level of basic services delivered by the municipalities. Questions were asked to obtain possible solutions to the problem. These challenges motivated the researcher to undertake a study to investigate the possible cause of poor delivery of service by municipalities to the communities. A possibility was to study and recommend a possible solution that would ensure that service is delivered on time, at the right price, at the right location and that quantity and quality meet the needs of the community. The system suitable to address this concern is demand management, the first element of the supply chain management system. The purpose of demand management is to ensure that the resources required to fulfil the needs identified during the strategic plan of the municipality, are delivered at the correct time, price and place, and that quantity and quality will satisfy the needs of the users. The research was conducted by means of a literature study and an empirical study. The literature study entails a literature discussion on three variables identified as crucial in achieving the objectives of the study, namely sustainability, demand management and change management. These variables form the basis of the evaluation of the factors which ensure that effectively used demand management results in sustainable service delivery. Delivering sustainable service will ensure that the service is delivered on time, at the right price, at the right location, and quality that meets the needs of the users. These variables were empirically tested in practise by means of a measurement instrument and subsequently evaluated. Based on the evaluation of the empirical study, the demographical data were analysed and concluded on. The reliability and internal consistence of the measurement instrument were determined by means of Cronbach Alpha coefficient. The results of the reliability test revealed that only change management’s Cronbach Alpha coefficient was less than the cut–off value of 0.7. Despite the results of change management’s Cronbach Alpha coefficient, the reliability of the other latent variables was found to be reliable and internally consistent. The variables were analysed and concluded upon by means of a descriptive statistical analysis, using frequency on the questionnaire responds. The arithmetic mean and standard deviation per construct was also calculated, using SPSS (Statistical Package for the Social Sciences). The relationship between the variables was also evaluated through determining the effect size and correlation of coefficient being calculated. Practical recommendations were suggested to address the weaknesses found during the evaluation of the questionnaires, in order to improve future continuity of both municipalities used as unit of measure. Recommendations of future research needed, were also made. / Thesis (MBA)--North-West University, Potchefstroom Campus, 2012.

Over the past months, newspapers and news bulletins are riddled by service delivery protests – to the extent that the President of the country acknowledges these protests. Many reasons were offered for the protests, but the main reason is cited as dissatisfaction with the level of basic services delivered by the municipalities. Questions were asked to obtain possible solutions to the problem. These challenges motivated the researcher to undertake a study to investigate the possible cause of poor delivery of service by municipalities to the communities. A possibility was to study and recommend a possible solution that would ensure that service is delivered on time, at the right price, at the right location and that quantity and quality meet the needs of the community. The system suitable to address this concern is demand management, the first element of the supply chain management system. The purpose of demand management is to ensure that the resources required to fulfil the needs identified during the strategic plan of the municipality, are delivered at the correct time, price and place, and that quantity and quality will satisfy the needs of the users. The research was conducted by means of a literature study and an empirical study. The literature study entails a literature discussion on three variables identified as crucial in achieving the objectives of the study, namely sustainability, demand management and change management. These variables form the basis of the evaluation of the factors which ensure that effectively used demand management results in sustainable service delivery. Delivering sustainable service will ensure that the service is delivered on time, at the right price, at the right location, and quality that meets the needs of the users. These variables were empirically tested in practise by means of a measurement instrument and subsequently evaluated. Based on the evaluation of the empirical study, the demographical data were analysed and concluded on. The reliability and internal consistence of the measurement instrument were determined by means of Cronbach Alpha coefficient. The results of the reliability test revealed that only change management’s Cronbach Alpha coefficient was less than the cut–off value of 0.7. Despite the results of change management’s Cronbach Alpha coefficient, the reliability of the other latent variables was found to be reliable and internally consistent. The variables were analysed and concluded upon by means of a descriptive statistical analysis, using frequency on the questionnaire responds. The arithmetic mean and standard deviation per construct was also calculated, using SPSS (Statistical Package for the Social Sciences). The relationship between the variables was also evaluated through determining the effect size and correlation of coefficient being calculated. Practical recommendations were suggested to address the weaknesses found during the evaluation of the questionnaires, in order to improve future continuity of both municipalities used as unit of measure. Recommendations of future research needed, were also made. / Thesis (MBA)--North-West University, Potchefstroom Campus, 2012.