abstract: Although the issue of factorial invariance has received increasing attention in the literature, the focus is typically on differences in factor structure across groups that are directly observed, such as those denoted by sex or ethnicity. While establishing factorial invariance across observed groups is a requisite step in making meaningful cross-group comparisons, failure to attend to possible sources of latent class heterogeneity in the form of class-based differences in factor structure has the potential to compromise conclusions with respect to observed groups and may result in misguided attempts at instrument development and theory refinement. The present studies examined the sensitivity of two widely used confirmatory factor analytic model fit indices, the chi-square test of model fit and RMSEA, to latent class differences in factor structure. Two primary questions were addressed. The first of these concerned the impact of latent class differences in factor loadings with respect to model fit in a single sample reflecting a mixture of classes. The second question concerned the impact of latent class differences in configural structure on tests of factorial invariance across observed groups. The results suggest that both indices are highly insensitive to class-based differences in factor loadings. Across sample size conditions, models with medium (0.2) sized loading differences were rejected by the chi-square test of model fit at rates just slightly higher than the nominal .05 rate of rejection that would be expected under a true null hypothesis. While rates of rejection increased somewhat when the magnitude of loading difference increased, even the largest sample size with equal class representation and the most extreme violations of loading invariance only had rejection rates of approximately 60%. RMSEA was also insensitive to class-based differences in factor loadings, with mean values across conditions suggesting a degree of fit that would generally be regarded as exceptionally good in practice. In contrast, both indices were sensitive to class-based differences in configural structure in the context of a multiple group analysis in which each observed group was a mixture of classes. However, preliminary evidence suggests that this sensitivity may contingent on the form of the cross-group model misspecification. / Dissertation/Thesis / Ph.D. Psychology 2011

A recent trend in education has been toward formative assessments to enable teachers, parents, and administrators assist students succeed. Cognitive diagnostic modeling (CDM) has the potential to provide valuable information for stakeholders to assist students identify their skill deficiency in specific academic subjects. Cognitive diagnosis models are mainly viewed as a family of latent class confirmatory probabilistic models. These models allow the mapping of students' skill profiles/academic ability. Using a complex simulation studies, the methodological issues in one of the existing cognitive models, referred to as compensatory reparameterized unified model (CRUM) under the log-linear model family of CDM, was investigated. In order for practitioners to implement these models, their item parameter recovery and examinees' classifications need to be studied in detail. A series of complex simulated data were generated for investigation with the following designs: three attributes with seven items, three attributes with thirty five items, four attributes with fifteen items, and five attributes with thirty one items. Each dataset was generated with observations of: 50, 100, 500, 1,000, 5,000, and 10,000 examinees. The first manuscript is the report of the investigation of how accurately CRUM could recover item parameters and classify examinees under true QMattrix specification and various research designs. The results suggested that the test length with regards to number of attributes and sample size affects the item parameter recovery and examinees classification accuracy. The second manuscript is the report of the investigation of the sensitivity of relative fit indices in detecting misfit for over- and opposite-Q-Matrix misspecifications. The relative fit indices under investigation were Akaike information criterion (AIC), Bayesian information criterion (BIC), and sample size adjusted Bayesian information criterion (ssaBIC). The results suggested that the CRUM can be a robust model given the consideration to the observation number and item/attribute combinations. The findings of this dissertation fill some of the existing gaps in the methodological issues regarding cognitive models' applicability and generalizability. It helps practitioners design tests in CDM framework in order to attain reliable and valid results. / Ph. D.

Most research on confirmatory factor analysis using global fit indices (AIC, BIC, AICc, and CAIC) has been in the structural equation modeling framework. Little research has been done concerning application of these indices to item response models, especially within the framework of multidimensional Rasch analysis. The results of two simulations studies that investigated how sample size, between-dimension correlation, and test length affect the accuracy of these indices in model recovery using a multidimensional Rasch analysis are described in this dissertation. The first study analyzed dichotomous data, with model-to-data misfit as an additional independent variable. The second study analyzed polytomous data, with rating scale structure as an additional independent variable. The interaction effect between global fit index and between-dimension correlation had very large effect sizes in both studies. At higher values of between-dimension correlation, AIC indicated the correct two-dimension generating structure slightly more often than does the BIC or CAIC. The correlation by test length interaction had an odds ratio indicating practical importance in the polytomous study but not the dichotomous study. The combination of shorter tests and higher correlations resulted in a difficult-to-detect distinction being modeled with less statistical information. The correlation by index interaction in the dichotomous study had an odds ratio indicating practical importance. As expected, the results demonstrated that violations of the Rasch model assumptions are magnified at higher between-dimension correlations. Recommendations for practitioners working with highly correlated multidimensional data include creating moderate length (roughly 40 items) instruments, minimizing data-to-model misfit in the choice of model used for confirmatory factor analysis (MRCMLM or other MIRT models), and making decisions based on multiple global indices instead of depending on one index in particular. / Ph. D.

The purpose of this research was to compare results from two approaches to measuring change over time. The multilevel model (MLM) and latent growth model (LGM) were imposed and the parameter estimates were compared, along with model fit. The study came out of education and used data collected from 191 teachers as part of a professional development intervention in science, which took place over four years. There were missing data as a result of teacher attrition. Teachers reported use of reform-oriented practices (ROP) was used as the outcome, and teacher-level variables were examined for their impact on initial ROP and change in ROP from baseline to one year after the intervention. Change in ROP was examined using a piecewise change model where two linear slopes were modeled. The first slope estimated the change from baseline to T1, or the initial change after the intervention while the second slope estimated the change from T1 to T3, or the secondary change. Parameter estimates obtained from MLM and LGM for a model using the error covariance structure commonly assumed in MLM (i.e., random slopes, homogeneous level-1 variance) were nearly identical. Models with various alternative covariance structures (commonly associated with the LGM framework) were examined, and results were nearly identical. Most of the model fit information was in agreement regarding the best fitting model being the model that assumed the typical MLM error covariance structure with the exception of the standardized root mean square residual (SRMR) fit index. The results from the models demonstrated that ROP increased after participating in the first year of the intervention and this level was sustained, though did not increase significantly in subsequent years. There was more variation in ROP at baseline. This information tells us that the intervention was successful in that after participating in the intervention the teachers' used ROP more frequently. The success of the intervention did not depend on any of the predictors that we assessed, and, as a group, the teachers became more similar in their use of reform-oriented practices over time.