The development of a standardized assessment can, in general, be broken into multiple stages. In the first, items to be used in the assessment are generated according to the skills and abilities that are to be assessed and the needs of the developers. These items are then, ideally, tested in the field on members of the population for which the assessment is intended. Item Response Theory (IRT) analysis is used to reveal items in the item pool which are unusable due to measurement error, redundancy in the level of item difficulty, or bias. More potential items may be generated and tested until there is a set of valid items with which the developers can move forward. The present study focused on the steps of item tryout and analysis for the establishment of demonstrable item-level validity. Fifty-one potential test items were analyzed for a version of the Shirts and Shoes Test (Plante & Vance, 2012) for 6-year-olds. A total of 23 items were discarded due to error in one or more of the measures mentioned above, and one item was discarded due to its low difficulty. The remaining 27 items were deemed suitable for the 6-year-old population.

Differential item functioning (DIF) is a psychometric issue routinely considered in educational and psychological assessment. However, it has not been studied in the context of a recently developed componential statistical model, the model with internal restrictions on item difficulty (MIRID; Butter, De Boeck, & Verhelst, 1998). Because the MIRID requires test questions measuring either single or multiple cognitive processes, it creates a complex environment for which traditional DIF methods may be inappropriate. This dissertation sought to extend the MIRID framework to detect DIF at the item-group level and the individual-item level. Such a model-based approach can increase the interpretability of DIF statistics by focusing on item characteristics as potential sources of DIF. In particular, group-level DIF may reveal comparative group strengths in certain secondary constructs. A simulation study was conducted to examine under different conditions parameter recovery, Type I error rates, and power of the proposed approach. Factors manipulated included sample size, magnitude of DIF, distributional characteristics of the groups, and the MIRID DIF models corresponding to discrete sources of differential functioning. The impact of studying DIF using wrong models was investigated. The results from the recovery study of the MIRID DIF model indicate that the four delta (i.e., non-zero value DIF) parameters were underestimated whereas item locations of the four associated items were overestimated. Bias and RMSE were significantly greater when delta was larger; larger sample size reduced RMSE substantially while the effects from the impact factor were neither strong nor consistent. Hypothesiswise and adjusted experimentwise Type I error rates were controlled in smaller delta conditions but not in larger delta conditions as estimates of zero-value DIF parameters were significantly different from zero. Detection power of the DIF model was weak. Estimates of the delta parameters of the three group-level DIF models, the MIRID differential functioning in components (DFFc), the MIRID differential functioning in item families (DFFm), and the MIRID differential functioning in component weights (DFW), were acceptable in general. They had good hypothesiswise and adjusted experimentwise Type I error control across all conditions and overall achieved excellent detection power. When fitting the proposed models to mismatched data, the false detection rates were mostly beyond the Bradley criterion because the zero-value DIF parameters in the mismatched model were not estimated adequately, especially in larger delta conditions. Recovery of item locations and component weights was also not adequate in larger delta conditions. Estimation of these parameters was more or less affected adversely by the DIF effect simulated in the mismatched data. To study DIF in MIRID data using the model-based approach, therefore, more research is necessary to determine the appropriate procedure or model to implement, especially for item-level differential functioning.

Achievement goal theory is one of the most broadly accepted theoretical paradigms in educational psychology with over 35 years of influencing research and educational practice. The longstanding use of this construct has led to two consequences of importance for this research: 1) many different dimensionality representations have been debated, and 2) methods used to confirm dimensionality of the scales have been supplanted from best practice. A further issue is that goal orientations are used to inform classroom practice, whereas most measurement studies focus on the structure of the personal goal orientation scales rather than the classroom level structure. This study aims to provide an updated understanding of one classroom goal orientation scale using the modern psychometric techniques of multidimensional item response theory and bifactor analysis. The most commonly used scale with K-12 students is the Patterns of Adaptive Learning Scales (PALS); thus, the PALS classroom goal orientation scales will be the subject of this study.

It is not uncommon to use unidimensional item response theory (IRT) models to estimate ability in multidimensional data. Therefore it is important to understand the implications of summarizing multiple dimensions of ability into a single parameter estimate, especially if effects are confounded when applied to computerized adaptive testing (CAT). Previous studies have investigated the effects of different IRT models and ability estimators by manipulating the relationships between item and person parameters. However, in all cases, the maximum information criterion was used as the item selection method. Because maximum information is heavily influenced by the item discrimination parameter, investigating a-stratified item selection methods is tenable. The current Monte Carlo study compared maximum information, a-stratification, and a-stratification with b blocking item selection methods, alone, as well as in combination with the Sympson-Hetter exposure control strategy. The six testing conditions were conditioned on three levels of interdimensional item difficulty correlations and four levels of interdimensional examinee ability correlations. Measures of fidelity, estimation bias, error, and item usage were used to evaluate the effectiveness of the methods. Results showed either stratified item selection strategy is warranted if the goal is to obtain precise estimates of ability when using unidimensional CAT in the presence of two-dimensional data. If the goal also includes limiting bias of the estimate, Sympson-Hetter exposure control should be included. Results also confirmed that Sympson-Hetter is effective in optimizing item pool usage. Given these results, existing unidimensional CAT implementations might consider employing a stratified item selection routine plus Sympson-Hetter exposure control, rather than recalibrate the item pool under a multidimensional model.

The Rey-Osterrieth Complex Figure Task (ROCFT) has been a standard in neuropsychological assessment for six decades. Many researchers have contributed administration procedures, additional scoring systems and normative data to improve its utility. Despite the abundance of research, the original 36-point scoring system still reigns among clinicians despite documented problems with ceiling and floor effects and poor discrimination between levels of impairment. This study is an attempt to provide a new method based upon item response theory that will allow clinicians to better describe the impairment levels of their patients. Through estimation of item characteristic curves, underlying traits can be estimated while taking into account varying levels of difficulty and discrimination within the set of individual items. The ultimate goal of the current research is identification of a subset of ROCFT items that can be examined in addition to total scores to provide an extra level of information for clinicians, particularly when they are faced with a need to discriminate severely and mildly impaired patients.

No description available.

The primary goal of this study was to develop a new procedure for estimating decision consistency and accuracy indices using the bifactor and testlet response theory (TRT) models. This study is the first to investigate decision consistency and accuracy from a multidimensional perspective, and the results have shown that the bifactor model at least behaved in way that met the author's expectations and represents a potential useful procedure. The TRT model, on the other hand, did not meet the author's expectations and generally showed poor model performance. The multidimensional decision consistency and accuracy indices proposed in this study appear to provide good performance, at least for the bifactor model, in the case of a substantial testlet effect. For practitioners examining a test containing testlets for decision consistency and accuracy, a recommended first step is to check for dimensionality. If the testlets show a significant degree of multidimensionality, then the usage of the multidimensional indices proposed can be recommended as the simulation study showed an improved level of performance over unidimensional IRT models. However, if there is a not a significant degree of multidimensionality then the unidimensional IRT models and indices would perform as well, or even better, than the multidimensional models. Another goal of this study was to compare methods for numerical integration used in the calculation of decision consistency and accuracy indices. This study investigated a new method (M method) that sampled ability estimates through a Monte-Carlo approach. In summary, the M method seems to be just as accurate as the other commonly used methods for numerical integration. However, it has some practical advantages over the D and P methods. As previously mentioned, it is not as nearly as computationally intensive as the D method. Also, the P method requires large sample sizes. In addition, the P method has conceptual disadvantage in that the conditioning variable, in theory, should be the true theta, not an estimated theta. The M method avoids both of these issues and seems to provide equally accurate estimates of decision consistency and accuracy indices, which makes it a strong option particularly in multidimensional cases.

No description available.

No description available.

With the novel data collection tools and diverse item types, computer-based assessments allow to easily obtain more information about an examinee’s response process such as response time (RT) data. This information has been utilized to increase the measurement precision about the latent ability in the response accuracy models. Van der Linden’s (2007) hierarchical speed-accuracy model has been widely used as a joint modelling framework to harness the information from RT and the response accuracy, simultaneously. The strict assumption of conditional independence between response and RT given latent ability and speed is commonly imposed in the joint modelling framework. Recently multiple studies (e.g., Bolsinova & Maris, 2016; Bolsinova, De Boeck, & Tijmstra, 2017a; Meng, Tao, & Chang, 2015) have found violations of the conditional independence assumption and proposed models to accommodate this violation by modelling conditional dependence of responses and RTs within a framework of Item Response Theory (IRT). Despite the widespread usage of Cognitive Diagnostic Models as formative assessment tools, the conditional joint modelling of responses and RTs has not yet been explored in this framework. Therefore, this research proposes a conditional joint response and RT model in CDM with an extended reparametrized higher-order deterministic input, noisy ‘and’ gate (DINA) model for the response accuracy. The conditional dependence is modelled by incorporating item-specific effects of residual RT (Bolsinova et al., 2017a) on the slope and intercept of the accuracy model. The effects of ignoring the conditional dependence on parameter recovery is explored with a simulation study, and empirical data analysis is conducted to demonstrate the application of the proposed model. Overall, modelling the conditional dependence, when applicable, has increased the correct attribute classification rates and resulted in more accurate item response parameter estimates.